math/test-tgmath3 test case size...

math/test-tgmath3 test case size...

[DJ Delorie]

In running the glibc testsuite on riscv, I discovered that one math
test (test-tgmath3) is, well, HUGE.  The generated .c file is 10 Mb.
This takes HOURS to compile on riscv, partly because of the huge, and
partly because on some targets where linker relaxation is the default,
link time can be O(size^2).

Even on my 4 GHz x86-64 just compiling the .o takes 50 seconds.

Can this test case be split up somehow?  Even just randomly spreading
it across a few files would be a huge benefit to test time.  Given the
number of tests in math/, "few" could mean "dozens"[*] and still be
reasonable :-)

Splitting it up allows for multi-threaded tests to spread the load
across build threads too, but in this case, even a single-threaded
build would benefit.

Also, if there's some way of scheduling that test *first*, at least it
could run in parallel with other tests, limiting the time at the end
when it's the only thing left running.  I.e. schedule longer tests
before shorter ones, so make can pack job slots more effectively.

(stdio-common/tst-printf-bz18872 suffers from this also, for slightly
different reasons, takes 9 sec on x86 and hours on riscv)

[*] Ideally it would mean "at least as many parts as the typical
    developer has cores to spread it across" ;-)

Re: math/test-tgmath3 test case size...

[Joseph Myers]

On Fri, 16 Feb 2018, DJ Delorie wrote:

> In running the glibc testsuite on riscv, I discovered that one math
> test (test-tgmath3) is, well, HUGE.  The generated .c file is 10 Mb.
> This takes HOURS to compile on riscv, partly because of the huge, and
> partly because on some targets where linker relaxation is the default,
> link time can be O(size^2).

That sounds like a good testcase to use for optimizing the linker.

That test is deliberately split up into a large number of small functions, 
called in a loop from a table listing those functions, to avoid 
(hopefully) any compiler performance issues that might arise from 
compiling large functions.

> Can this test case be split up somehow?  Even just randomly spreading
> it across a few files would be a huge benefit to test time.  Given the
> number of tests in math/, "few" could mean "dozens"[*] and still be
> reasonable :-)

It could easily enough be split up by function (list the functions in the 
makefile, generate a .c file for each function, include a test that 
verifies that the list of functions in the makefile does match the list in 
the script itself of functions for which it can generate tests).  Tests of 
fma make up nearly a third of the total, however (since that has the most 
type-generic arguments).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: math/test-tgmath3 test case size...

[DJ Delorie]

Joseph Myers <joseph@codesourcery.com> writes:
> That sounds like a good testcase to use for optimizing the linker.

Well, *any* large object is that :-)

> It could easily enough be split up by function (list the functions in the 
> makefile, generate a .c file for each function, include a test that 
> verifies that the list of functions in the makefile does match the list in 
> the script itself of functions for which it can generate tests).  Tests of 
> fma make up nearly a third of the total, however (since that has the most 
> type-generic arguments).

Could the generating script take a parameter N that says "emit every Nth
function" and just generate, say, 10 files?  I.e. run "./generate 1
test-tgmath3-1.c" then "./generate 2 test-tgmath3-2.c" etc ?

It doesn't have to be a meaningful split.

Re: math/test-tgmath3 test case size...

[Joseph Myers]

On Fri, 16 Feb 2018, DJ Delorie wrote:

> > It could easily enough be split up by function (list the functions in the 
> > makefile, generate a .c file for each function, include a test that 
> > verifies that the list of functions in the makefile does match the list in 
> > the script itself of functions for which it can generate tests).  Tests of 
> > fma make up nearly a third of the total, however (since that has the most 
> > type-generic arguments).
> 
> Could the generating script take a parameter N that says "emit every Nth
> function" and just generate, say, 10 files?  I.e. run "./generate 1
> test-tgmath3-1.c" then "./generate 2 test-tgmath3-2.c" etc ?

You'd be effectively generating all the tests 10 times and throwing most 
of them away every time (though since the script takes under a second to 
run, that may not be too much of a problem), whereas when generating 73 
files, one for each of the 73 macros being tested, you could easily make 
gen-tgmath-tests.py not do the work for the 72 macros it's not been asked 
to generate tests for.  (Also, I'd rather that test names are meaningfully 
stable rather than having test-tgmath3-1 mean something completely 
different every time something about this test changes - though in this 
case, either all of test-tgmath3 should pass, or there should be lots of 
failures if there's been a buggy change to tgmath.h.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: math/test-tgmath3 test case size...

[DJ Delorie]

So I did some statistics on test-tgmath3.c.  There are 73 functions
tested, but the number of tests-per-function depend on the number of
arguments to that function, since we test permutations of all arguments.
There are 25 types tested for riscv.

Thus, fma(), which takes three arguments, accounts for 15,625 of the
28,823 tests (15,625 = 25*25*25).  So even if we split tgmath3 up by
function name, we're still left with one source that will be huge (the
rest are: one 2200, a few 625, the rest under 50 tests-per-function).

Splitting by return type doesn't help, as 15,849 of the tests return
"double" (no surprise).

If we split by the type of the first argument, we get 47 files of at
most 1,097 tests per file.  This is a good split for compiling, but not
an intuitive split, and the Makefile rules for it might be confusing or
tricky.

In theory, at least, if we split out fma, we'd have two files each of
which could compile in 1/4 the time (assuming the O(N^2) affects are
primary), and could run on two cores, resuling in 1/8th the wall clock
time.  I might test this if I can improve my python-fu and find time on
the simulator.

I'll also ask the obvious "why so many permutations" but I assume the
answer is "because testing" ;-)

Re: math/test-tgmath3 test case size...

[Carlos O'Donell]

On 02/19/2018 01:29 PM, DJ Delorie wrote:
> 
> So I did some statistics on test-tgmath3.c.  There are 73 functions
> tested, but the number of tests-per-function depend on the number of
> arguments to that function, since we test permutations of all arguments.
> There are 25 types tested for riscv.
> 
> Thus, fma(), which takes three arguments, accounts for 15,625 of the
> 28,823 tests (15,625 = 25*25*25).  So even if we split tgmath3 up by
> function name, we're still left with one source that will be huge (the
> rest are: one 2200, a few 625, the rest under 50 tests-per-function).
> 
> Splitting by return type doesn't help, as 15,849 of the tests return
> "double" (no surprise).
> 
> If we split by the type of the first argument, we get 47 files of at
> most 1,097 tests per file.  This is a good split for compiling, but not
> an intuitive split, and the Makefile rules for it might be confusing or
> tricky.
> 
> In theory, at least, if we split out fma, we'd have two files each of
> which could compile in 1/4 the time (assuming the O(N^2) affects are
> primary), and could run on two cores, resuling in 1/8th the wall clock
> time.  I might test this if I can improve my python-fu and find time on
> the simulator.
> 
> I'll also ask the obvious "why so many permutations" but I assume the
> answer is "because testing" ;-)
> 

Because testing.

Splitting out fma seems like it might be a good idea.

-- 
Cheers,
Carlos.