libm-test summary confusion

libm-test summary confusion

[Joseph Simmons-Talbott]

I've been digging into the libm-test driver and tests and am confused
on a point.  The summary for math/test-double-acos on my system
(x86_64) shows:

Test suite completed:
  400 test cases plus 396 tests for exception flags and
    396 tests for errno executed.
  All tests passed successfully.

The wording seems to imply that 1192 tests were run.  From digging
through the code I believe it might be better to replace the word
"plus" with something like "of which".  But my larger confusion comes
from the 400 value which from my understanding of the code come from
the call to update_stats() in print_max_error() which is called by the
END macro for each of the 4 rounding modes.  From instrumenting the
code I've determined that there are 99 entries in acos_test_data in
the generated libm-test-acos.c file and a run for each of the 4
rounding modes would yield the expected 396 and the calls to
print_max_error() for each rounding mode yields the 400. Another
possibility is that I am missing a distinction between "test cases"
and "tests" but that seems less likely.

Can anyone please help me to understand this better?

Thanks,
Joe

Re: libm-test summary confusion

[Adhemerval Zanella Netto]

On 02/05/24 14:45, Joseph Simmons-Talbott via Libc-help wrote:
> I've been digging into the libm-test driver and tests and am confused
> on a point.  The summary for math/test-double-acos on my system
> (x86_64) shows:
> 
> Test suite completed:
>   400 test cases plus 396 tests for exception flags and
>     396 tests for errno executed.
>   All tests passed successfully.
> 
> The wording seems to imply that 1192 tests were run.  From digging
> through the code I believe it might be better to replace the word
> "plus" with something like "of which".  But my larger confusion comes
> from the 400 value which from my understanding of the code come from
> the call to update_stats() in print_max_error() which is called by the
> END macro for each of the 4 rounding modes.  From instrumenting the
> code I've determined that there are 99 entries in acos_test_data in
> the generated libm-test-acos.c file and a run for each of the 4
> rounding modes would yield the expected 396 and the calls to
> print_max_error() for each rounding mode yields the 400. Another
> possibility is that I am missing a distinction between "test cases"
> and "tests" but that seems less likely.
> 
> Can anyone please help me to understand this better?

Maybe we can break down the lines with a more descriptive sentence,
where it states that the execution tests cover both exception and
errno checks.  Something like:

  Test suite completed: total of 400 executed tests:
  * 4 tests for maximum expected error in all support rounding modes
  * 396 execution tests where:
    - 396 tests for exception flags
    - 396 tests for errno return code
    
(maybe we can improve the 'execution tests' wording, not sure)

Re: libm-test summary confusion

[Carlos O'Donell]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

On Mon, May 6, 2024, 15:35 Adhemerval Zanella Netto via Libc-help, <
libc-help@sourceware.org> wrote:

>
>
> On 02/05/24 14:45, Joseph Simmons-Talbott via Libc-help wrote:
> > I've been digging into the libm-test driver and tests and am confused
> > on a point.  The summary for math/test-double-acos on my system
> > (x86_64) shows:
> >
> > Test suite completed:
> >   400 test cases plus 396 tests for exception flags and
> >     396 tests for errno executed.
> >   All tests passed successfully.
> >
> > The wording seems to imply that 1192 tests were run.  From digging
> > through the code I believe it might be better to replace the word
> > "plus" with something like "of which".  But my larger confusion comes
> > from the 400 value which from my understanding of the code come from
> > the call to update_stats() in print_max_error() which is called by the
> > END macro for each of the 4 rounding modes.  From instrumenting the
> > code I've determined that there are 99 entries in acos_test_data in
> > the generated libm-test-acos.c file and a run for each of the 4
> > rounding modes would yield the expected 396 and the calls to
> > print_max_error() for each rounding mode yields the 400. Another
> > possibility is that I am missing a distinction between "test cases"
> > and "tests" but that seems less likely.
> >
> > Can anyone please help me to understand this better?
>
> Maybe we can break down the lines with a more descriptive sentence,
> where it states that the execution tests cover both exception and
> errno checks.  Something like:
>
>   Test suite completed: total of 400 executed tests:
>   * 4 tests for maximum expected error in all support rounding modes
>   * 396 execution tests where:
>     - 396 tests for exception flags
>     - 396 tests for errno return code
>
> (maybe we can improve the 'execution tests' wording, not sure)
>

I agree 100% and I sent Joe a prototype patch privately that does something
very similar to what you suggest.

I called the 4 tests "builtin" tests, and I added a distinct count for
exception and rounding modes to accurately count how many of each are run.

I would be happy to see this cleaned up.

Cheers,
Carlos.