How to skip test when building glibc

How to skip test when building glibc

[Letu Ren]

Hi,

I want to build glibc on ArchLinux RISC-V, however, some unit tests
failure block my build script. I noticed that those failure also
appeared in official release notes of glibc v2.35. So, I think it's
safe to skip them. The problem is how to skip specific unit tests.
glibc uses autotools as the build system. However, I cannot find TESTS
and XFAIL_TESTS variables in Makefile. Glibc wiki of testsuite doesn't
mention it as well. I tried adding the XFAIL_TESTS variable to skip
tests, however, I failed. Could anyone help on this issue?

Letu Ren

Re: How to skip test when building glibc

[Adhemerval Zanella]

> On 10 Jun 2022, at 23:49, Letu Ren via Libc-help <libc-help@sourceware.org> wrote:
> 
> Hi,
> 
> I want to build glibc on ArchLinux RISC-V, however, some unit tests
> failure block my build script. I noticed that those failure also
> appeared in official release notes of glibc v2.35. So, I think it's
> safe to skip them. The problem is how to skip specific unit tests.
> glibc uses autotools as the build system. However, I cannot find TESTS
> and XFAIL_TESTS variables in Makefile. Glibc wiki of testsuite doesn't
> mention it as well. I tried adding the XFAIL_TESTS variable to skip
> tests, however, I failed. Could anyone help on this issue?
> 
> Letu Ren


It depends of what you are doing on your build script. The ‘make’ rule does
not run unit unit test, so if you intend to just build glibc it should be suffice.

However, if you are trying to run the expected make and make check you
try to not run the tests itself with the extra ‘run-built-tests=no’ rule with
‘make check’.

Unfortunately, there is no way to avoid specific tests in a default 'make check’
and this is intentional: a failure really should represent an issue that might be
investigated. 

We are trying to either get rid of flaky tests or improve them in a way to make
them report robust output, to avoid environment issues that might mislead
the runner.

What kind of issue are you seeing in you environment?

Re: How to skip test when building glibc

[Letu Ren]

Hi,

> It depends of what you are doing on your build script. The ‘make’ rule does
> not run unit unit test, so if you intend to just build glibc it should be suffice.

I'm currently trying to build glibc for ArchLinux RISC-V. My build
script executes `make` then `make check`. "It is highly recommended to
have check() as it helps to make sure software has been built
correctly and works fine with its dependencies." according to
ArchLinux wiki Creating packages. So I think I need to `make check`.

> However, if you are trying to run the expected make and make check you
> try to not run the tests itself with the extra ‘run-built-tests=no’ rule with
> ‘make check’.

As I'm new to libc, I'm not sure what `run-built-tests=no` does
exactly. I cannot find a wiki page which mentions this make variable.
I found some mail archives about it. "If you look at some of the
Makefiles in glibc, it conditionally includes/excludes some tests from
being built based on the value of run-built-tests.". And according to
glibc/Makeconfig, "Whether to run test programs built for the
library's host system." I wonder what this variable does exactly and
whether `make check run-built-tests=no` can make sure glibc built by
myself is able to work correctly.

> Unfortunately, there is no way to avoid specific tests in a default 'make check’
> and this is intentional: a failure really should represent an issue that might be
> investigated.
>
> We are trying to either get rid of flaky tests or improve them in a way to make
> them report robust output, to avoid environment issues that might mislead
> the runner.
>
> What kind of issue are you seeing in you environment?

I encountered 16 test failures. 11 of them are due to timeouts.
FAIL: nss/tst-nss-files-hosts-getent
FAIL: nss/tst-nss-files-hosts-multi
FAIL: string/test-memcpy
FAIL: string/test-memcpy-large
FAIL: string/test-mempcpy
FAIL: stdio-common/tst-vfprintf-width-prec
FAIL: stdio-common/tst-vfprintf-width-prec-mem
FAIL: resolv/tst-resolv-res_init-multi
FAIL: malloc/tst-malloc-too-large-malloc-hugetlb2
FAIL: nptl/tst-mutex10
FAIL: locale/tst-localedef-path-norm

 And Five of them are mentioned in release notes of glibc v2.35. So I
think those failures are expected.
FAIL: math/test-float-j0
FAIL: math/test-float32-j0
FAIL: stdlib/tst-strfrom
FAIL: stdlib/tst-strfrom-locale
FAIL: stdlib/test-bz22786

As far as I know, ArchLinux officially skips some tests using  sed in
Makefile. Just simply remove some tests. In my case, I cannot simply
sed, because I cannot find test-float-j0, which is generated in
runtime. I planned to execute `TIMEOUTFACTOR=300 make check` to make
sure no timeout issue occurs. So the only problem is how to deal with
five expected to fail tests. Could you please give me some advice?

Re: How to skip test when building glibc

[Florian Weimer]

* Letu Ren via Libc-help:

> I encountered 16 test failures. 11 of them are due to timeouts.
> FAIL: nss/tst-nss-files-hosts-getent
> FAIL: nss/tst-nss-files-hosts-multi
> FAIL: string/test-memcpy
> FAIL: string/test-memcpy-large
> FAIL: string/test-mempcpy
> FAIL: stdio-common/tst-vfprintf-width-prec
> FAIL: stdio-common/tst-vfprintf-width-prec-mem
> FAIL: resolv/tst-resolv-res_init-multi
> FAIL: malloc/tst-malloc-too-large-malloc-hugetlb2
> FAIL: nptl/tst-mutex10
> FAIL: locale/tst-localedef-path-norm

Most of them occur regularly on very slow systems (but I haven't seen
nptl/tst-mutex10 and malloc/tst-malloc-too-large-malloc-hugetlb2 in this
context).  You could setting TIMEOUT_FACTOR to a large value (like 100)
and retry.

Thanks,
Florian

Re: How to skip test when building glibc

[Letu Ren]

On Wed, Jun 15, 2022 at 8:11 PM Florian Weimer <fweimer@redhat.com> wrote:
> Most of them occur regularly on very slow systems (but I haven't seen
> nptl/tst-mutex10 and malloc/tst-malloc-too-large-malloc-hugetlb2 in this
> context).  You could setting TIMEOUT_FACTOR to a large value (like 100)
> and retry.

Yes, I have figured out the solution to the timeout issue. Do you know
how to deal with other failures? Thanks for replying anyway.

Letu Ren

Re: How to skip test when building glibc

[Adhemerval Zanella]

> On 15 Jun 2022, at 05:51, Letu Ren <fantasquex@gmail.com> wrote:
> 
> On Wed, Jun 15, 2022 at 8:11 PM Florian Weimer <fweimer@redhat.com> wrote:
>> Most of them occur regularly on very slow systems (but I haven't seen
>> nptl/tst-mutex10 and malloc/tst-malloc-too-large-malloc-hugetlb2 in this
>> context).  You could setting TIMEOUT_FACTOR to a large value (like 100)
>> and retry.
> 
> Yes, I have figured out the solution to the timeout issue. Do you know
> how to deal with other failures? Thanks for replying anyway.
> 
> Letu Ren

I think ideally we should either split the tests that are taking too much time
or increase the minimum expected timeout (so TIMEOUT_FACTOR is not
really needed as a default option).

For the tests that haven’t failed due a TIMEOUT, which are the *.out contents?

Re: How to skip test when building glibc

[Letu Ren]

On Thu, Jun 16, 2022 at 4:33 AM Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
>
> I think ideally we should either split the tests that are taking too much time
> or increase the minimum expected timeout (so TIMEOUT_FACTOR is not
> really needed as a default option).

Yes, I totally agree with you. I think I should only increase the
timeout limit to those tests which are necessary. Setting
TIMEOUTFACTOR of all tests to 300 is not a good idea. However I don't
know how to do that. I haven't figured out where the timeout limit is
set after looking through code and doc. Should I report those failures
to glibc bug mailing list, because some of them aren't mentioned in
release notes?

> For the tests that haven’t failed due a TIMEOUT, which are the *.out contents?

stdlib/test-bz22786 is another failure due to a timeout issue, It's
just my carelessness. So we have four failures left. Their *.out
contents are listed below.

# cat math/test-float-j0.out
testing float (without inline functions)
Failure: Test: j0_upward (0x2.67a2a4p+0)
Result:
is:          5.64343701e-08   0x1.e4c47ep-25
should be:   5.64344021e-08   0x1.e4c490p-25
difference:  3.19744232e-14   0x1.200000p-45
ulp       :  9.0000
max.ulp   :  8.0000
Maximal error of `j0_upward'
is      : 9 ulp
accepted: 8 ulp

Test suite completed:
 348 test cases plus 344 tests for exception flags and
   344 tests for errno executed.
 2 errors occurred.

# cat math/test-float32-j0.out
testing _Float32 (without inline functions)
Failure: Test: j0_upward (0x2.67a2a4p+0)
Result:
is:          5.64343701e-08   0x1.e4c47ep-25
should be:   5.64344021e-08   0x1.e4c490p-25
difference:  3.19744232e-14   0x1.200000p-45
ulp       :  9.0000
max.ulp   :  8.0000
Maximal error of `j0_upward'
is      : 9 ulp
accepted: 8 ulp

Test suite completed:
 348 test cases plus 344 tests for exception flags and
   344 tests for errno executed.
 2 errors occurred.

# cat stdlib/tst-strfrom.out
Testing in locale: C
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)
Testing in locale: en_US.ISO-8859-1
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)
Testing in locale: en_US.UTF-8
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)

# cat stdlib/tst-strfrom-locale.out
Testing in locale: de_DE.UTF-8
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)
Testing in locale: tr_TR.ISO-8859-9
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)
Testing in locale: tr_TR.UTF-8
strfromf: got NAN (3), expected -NAN (4)
strfromf32: got NAN (3), expected -NAN (4)

Re: How to skip test when building glibc

[Adhemerval Zanella]

> On 15 Jun 2022, at 22:13, Letu Ren <fantasquex@gmail.com> wrote:
> 
> On Thu, Jun 16, 2022 at 4:33 AM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
>> 
>> I think ideally we should either split the tests that are taking too much time
>> or increase the minimum expected timeout (so TIMEOUT_FACTOR is not
>> really needed as a default option).
> 
> Yes, I totally agree with you. I think I should only increase the
> timeout limit to those tests which are necessary. Setting
> TIMEOUTFACTOR of all tests to 300 is not a good idea. However I don't
> know how to do that. I haven't figured out where the timeout limit is
> set after looking through code and doc. Should I report those failures
> to glibc bug mailing list, because some of them aren't mentioned in
> release notes?

Yes I think it worth to set a higher bond for tests that require a lot of time,
the issue is usually we tests with faster machines where defaults are good
enough. If you may please open bugs with the required time on the platform 
you are testing, it would require to adjust on each test code to increase the
required timeout.

> 
>> For the tests that haven’t failed due a TIMEOUT, which are the *.out contents?
> 
> stdlib/test-bz22786 is another failure due to a timeout issue, It's
> just my carelessness. So we have four failures left. Their *.out
> contents are listed below.
> 
> # cat math/test-float-j0.out
> testing float (without inline functions)
> Failure: Test: j0_upward (0x2.67a2a4p+0)
> Result:
> is:          5.64343701e-08   0x1.e4c47ep-25
> should be:   5.64344021e-08   0x1.e4c490p-25
> difference:  3.19744232e-14   0x1.200000p-45
> ulp       :  9.0000
> max.ulp   :  8.0000
> Maximal error of `j0_upward'
> is      : 9 ulp
> accepted: 8 ulp
> 
> Test suite completed:
> 348 test cases plus 344 tests for exception flags and
>   344 tests for errno executed.
> 2 errors occurred.
> 
> # cat math/test-float32-j0.out
> testing _Float32 (without inline functions)
> Failure: Test: j0_upward (0x2.67a2a4p+0)
> Result:
> is:          5.64343701e-08   0x1.e4c47ep-25
> should be:   5.64344021e-08   0x1.e4c490p-25
> difference:  3.19744232e-14   0x1.200000p-45
> ulp       :  9.0000
> max.ulp   :  8.0000
> Maximal error of `j0_upward'
> is      : 9 ulp
> accepted: 8 ulp
> 
> Test suite completed:
> 348 test cases plus 344 tests for exception flags and
>   344 tests for errno executed.
> 2 errors occurred.

This might be a bug either in the compiler or in the environment you are using.
The commit log for riscv32 libm-test-ulps (b2d175cdb755277ef5579fdac914768003bfbc5c) 
states that values does not match the RV32 on QEMU, so it might what is
happening.

If you are running on real hardware, it might be case where floating-point
in non default rounding mode has higher ULPs than expected (I am assuming
the RV64 ones were done in read hardware). You can
regenerate the libm-test-ulps with `make regen-ulp`, this will show some
information on how to copy back the file on source tree.

We will need to increase the generic threshold what we consider an error
(as you can see any math function that shows ULPs larger than 8). 

> 
> # cat stdlib/tst-strfrom.out
> Testing in locale: C
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: en_US.ISO-8859-1
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: en_US.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> 
> # cat stdlib/tst-strfrom-locale.out
> Testing in locale: de_DE.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: tr_TR.ISO-8859-9
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)
> Testing in locale: tr_TR.UTF-8
> strfromf: got NAN (3), expected -NAN (4)
> strfromf32: got NAN (3), expected -NAN (4)

This is more worrisome because it means the NAN signal is not being
propagated by some FP operation. I think you will need to debug to 
understand better that is happening, since strfromf code does a lot of FP 
operations.

Re: How to skip test when building glibc

[Letu Ren]

> Yes I think it worth to set a higher bond for tests that require a lot of time,
> the issue is usually we tests with faster machines where defaults are good
> enough. If you may please open bugs with the required time on the platform
> you are testing, it would require to adjust on each test code to increase the
> required timeout.

I think I should report bugs to glibc. The problem is where to know
the time limit for each failed unit test set by glibc and how to
measure the time consumed by tests. I think using bash time is not
precise. Is there something I can refer to?

> This might be a bug either in the compiler or in the environment you are using.
> The commit log for riscv32 libm-test-ulps (b2d175cdb755277ef5579fdac914768003bfbc5c)
> states that values does not match the RV32 on QEMU, so it might what is
> happening.
>
> If you are running on real hardware, it might be case where floating-point
> in non default rounding mode has higher ULPs than expected (I am assuming
> the RV64 ones were done in read hardware). You can
> regenerate the libm-test-ulps with `make regen-ulp`, this will show some
> information on how to copy back the file on source tree.
>
> We will need to increase the generic threshold what we consider an error
> (as you can see any math function that shows ULPs larger than 8).

I use a HiFive Unmatched board to test. The hardware baseline is
RV64GC and the ABI is lp64d. I don't think I have the ability to solve
this problem currently. May need some time to learn how libm works. :(

> This is more worrisome because it means the NAN signal is not being
> propagated by some FP operation. I think you will need to debug to
> understand better that is happening, since strfromf code does a lot of FP
> operations.

I don't think I have the ability to solve this problem currently. :(
Anyway, thanks for replying and analyzing the reason why unit tests
fail.  I will try to investigate them.

Letu Ren

Re: How to skip test when building glibc

[Adhemerval Zanella]

> On 17 Jun 2022, at 05:12, Letu Ren <fantasquex@gmail.com> wrote:
> 
>> Yes I think it worth to set a higher bond for tests that require a lot of time,
>> the issue is usually we tests with faster machines where defaults are good
>> enough. If you may please open bugs with the required time on the platform
>> you are testing, it would require to adjust on each test code to increase the
>> required timeout.
> 
> I think I should report bugs to glibc. The problem is where to know
> the time limit for each failed unit test set by glibc and how to
> measure the time consumed by tests. I think using bash time is not
> precise. Is there something I can refer to?

The default configuration for all testcases is to create a helper parent 
process to handle timeout and spurious failures (such as segfault). You 
can disable it by using the —direct command line with testrun.sh (which 
runs the test using the built glibc loader and libraries):

  builddir$ time ./testrun testcase —direct

Some tests might require additional environment variables or arguments,
easiest way to check is to get them is to copy from make check log.

> 
>> This might be a bug either in the compiler or in the environment you are using.
>> The commit log for riscv32 libm-test-ulps (b2d175cdb755277ef5579fdac914768003bfbc5c)
>> states that values does not match the RV32 on QEMU, so it might what is
>> happening.
>> 
>> If you are running on real hardware, it might be case where floating-point
>> in non default rounding mode has higher ULPs than expected (I am assuming
>> the RV64 ones were done in read hardware). You can
>> regenerate the libm-test-ulps with `make regen-ulp`, this will show some
>> information on how to copy back the file on source tree.
>> 
>> We will need to increase the generic threshold what we consider an error
>> (as you can see any math function that shows ULPs larger than 8).
> 
> I use a HiFive Unmatched board to test. The hardware baseline is
> RV64GC and the ABI is lp64d. I don't think I have the ability to solve
> this problem currently. May need some time to learn how libm works. :(

So you are indeed testing against the real hardware.  Could you open a
bug report [1] with the output of ‘make regen-ulps’ ? I think we will
need to update the minimum required ULPs.

It can be also a regression on compiler that is generating larger error
bounds, since the current ULP baseline seems also to be generated on
read hardware.

> 
>> This is more worrisome because it means the NAN signal is not being
>> propagated by some FP operation. I think you will need to debug to
>> understand better that is happening, since strfromf code does a lot of FP
>> operations.
> 
> I don't think I have the ability to solve this problem currently. :(
> Anyway, thanks for replying and analyzing the reason why unit tests
> fail.  I will try to investigate them.

Could you also open another bug report with the output of the testcases?

> 
> Letu Ren

[1] https://sourceware.org/bugzilla/ <https://sourceware.org/bugzilla/>

Re: How to skip test when building glibc

[Letu Ren]

Hi,

First, I'm really sorry for my late reply.

> The default configuration for all testcases is to create a helper parent
> process to handle timeout and spurious failures (such as segfault). You
> can disable it by using the —direct command line with testrun.sh (which
> runs the test using the built glibc loader and libraries):
>
>   builddir$ time ./testrun testcase —direct
>
> Some tests might require additional environment variables or arguments,
> easiest way to check is to get them is to copy from make check log.

Thanks a lot. Now I know how to calculate time consumed by each failing test.

> So you are indeed testing against the real hardware.  Could you open a
> bug report [1] with the output of ‘make regen-ulps’ ? I think we will
> need to update the minimum required ULPs.
>
> It can be also a regression on compiler that is generating larger error
> bounds, since the current ULP baseline seems also to be generated on
> read hardware.

As glibc 2.36 was released several days ago, I rerun testsuite with
glibc 2.36. This issue is fixed.

> Could you also open another bug report with the output of the testcases?

OK, I have filed several bug reports to bugzilla.
https://sourceware.org/bugzilla/show_bug.cgi?id=29501
https://sourceware.org/bugzilla/show_bug.cgi?id=29500
https://sourceware.org/bugzilla/show_bug.cgi?id=29499

Thank you very much for your kind help.

Letu Ren