Re: How to skip test when building glibc

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

> On 15 Jun 2022, at 22:13, Letu Ren <fantasquex@gmail.com> wrote:
> 
> On Thu, Jun 16, 2022 at 4:33 AM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
>> 
>> I think ideally we should either split the tests that are taking too much time
>> or increase the minimum expected timeout (so TIMEOUT_FACTOR is not
>> really needed as a default option).
> 
> Yes, I totally agree with you. I think I should only increase the
> timeout limit to those tests which are necessary. Setting
> TIMEOUTFACTOR of all tests to 300 is not a good idea. However I don't
> know how to do that. I haven't figured out where the timeout limit is
> set after looking through code and doc. Should I report those failures
> to glibc bug mailing list, because some of them aren't mentioned in
> release notes?

Yes I think it worth to set a higher bond for tests that require a lot of time,
the issue is usually we tests with faster machines where defaults are good
enough. If you may please open bugs with the required time on the platform 
you are testing, it would require to adjust on each test code to increase the
required timeout.

> 
>> For the tests that haven’t failed due a TIMEOUT, which are the *.out contents?
> 
> stdlib/test-bz22786 is another failure due to a timeout issue, It's
> just my carelessness. So we have four failures left. Their *.out
> contents are listed below.
> 
> # cat math/test-float-j0.out
> testing float (without inline functions)
> Failure: Test: j0_upward (0x2.67a2a4p+0)
> Result:
> is:          5.64343701e-08   0x1.e4c47ep-25
> should be:   5.64344021e-08   0x1.e4c490p-25
> difference:  3.19744232e-14   0x1.200000p-45
> ulp       :  9.0000
> max.ulp   :  8.0000
> Maximal error of `j0_upward'
> is      : 9 ulp
> accepted: 8 ulp
> 
> Test suite completed:
> 348 test cases plus 344 tests for exception flags and
>   344 tests for errno executed.
> 2 errors occurred.
> 
> # cat math/test-float32-j0.out
> testing _Float32 (without inline functions)
> Failure: Test: j0_upward (0x2.67a2a4p+0)
> Result:
> is:          5.64343701e-08   0x1.e4c47ep-25
> should be:   5.64344021e-08   0x1.e4c490p-25
> difference:  3.19744232e-14   0x1.200000p-45
> ulp       :  9.0000
> max.ulp   :  8.0000
> Maximal error of `j0_upward'
> is      : 9 ulp
> accepted: 8 ulp
> 
> Test suite completed:
> 348 test cases plus 344 tests for exception flags and
>   344 tests for errno executed.
> 2 errors occurred.

This might be a bug either in the compiler or in the environment you are using.
The commit log for riscv32 libm-test-ulps (b2d175cdb755277ef5579fdac914768003bfbc5c) 
states that values does not match the RV32 on QEMU, so it might what is
happening.

If you are running on real hardware, it might be case where floating-point
in non default rounding mode has higher ULPs than expected (I am assuming
the RV64 ones were done in read hardware). You can
regenerate the libm-test-ulps with `make regen-ulp`, this will show some
information on how to copy back the file on source tree.

We will need to increase the generic threshold what we consider an error
(as you can see any math function that shows ULPs larger than 8). 

> 
> # cat stdlib/tst-strfrom.out
> Testing in locale: C
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: en_US.ISO-8859-1
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: en_US.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> 
> # cat stdlib/tst-strfrom-locale.out
> Testing in locale: de_DE.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: tr_TR.ISO-8859-9
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: tr_TR.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)

This is more worrisome because it means the NAN signal is not being
propagated by some FP operation. I think you will need to debug to 
understand better that is happening, since strfromf code does a lot of FP 
operations.