Re: glibc test procedure needed

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

> On 8 Jul 2022, at 01:49, sundeep.kokkonda@gmail.com wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>> Sent: 05 July 2022 18:13
>> To: sundeep.kokkonda@gmail.com
>> Cc: Libc-help <libc-help@sourceware.org>
>> Subject: Re: glibc test procedure needed
>> 
>> 
>> 
>>> On 5 Jul 2022, at 03:09, sundeep.kokkonda--- via Libc-alpha <libc-
>> alpha@sourceware.org> wrote:
>>> 
>>> Hello,
>>> 
>>> 
>>> 
>>> I need some info on glibc testing.
>>> 
>>> 
>>> 
>>> In the Yocto project we're updating glibc frequently and so we need to
>>> ensure the updated glibc is not introduced any issues, by performing
>>> the testing.
>>> 
>>> Can you let me know how can I test glibc? Is there any standardized
>>> test procedure available to do full test/regression test?
>>> 
>> 
>> The usual ‘make check’ should cover all the required testing, you might need to
>> use ‘run-built-tests=yes’ additional directive if you are cross-compiling to force
>> the make to issues the built tests (for instance on architecture that support
>> multiple ABI, like x86, or to force the use of some emulation like qemu-user).
>> 
>> We are still fixing up some clunky tests and there are some broken environment
>> (for instance broken syscall filtering) where some tests might advertise a failure.
> 
> Hello,
> 
> I tried with 'Testing with a cross-compiler' method and test is aborted when the flag '-Wformat-security' is enabled. When this flag is removed the test is executed successfully. Is it a known issue? (The error log make-check.out is attached for reference)
> Also, can anyone share the list of known test failures (if any).
> 
> 
> Thanks,
> Sundeep K.
> 
> <make-check.out>


Since glibc enables -Werror as default, not all compiler flags are supported
without triggering warning that might abort the build.  Ideally for each new
warning flag you will first build with —disable-error and check each error
to possible either check if this is a compiler issue or if it requires patch
glibc.

For the ‘make check’ output you can see why might be happening by checking
each file.out file:

UNSUPPORTED: assert/tst-assert-c++
UNSUPPORTED: assert/tst-assert-g++

It means that you do not configure with a C++ compiler (it is required only
for testing).

FAIL: catgets/test1.cat

I am not sure about this one, but it seems that is a locale not being present
in the testing.

FAIL: conform/ISO/setjmp.h/conform
FAIL: conform/ISO/stdio.h/conform
FAIL: conform/ISO/stdlib.h/conform
FAIL: conform/ISO/stdlib.h/linknamespace
FAIL: conform/ISO11/setjmp.h/conform
FAIL: conform/ISO11/stdio.h/conform
FAIL: conform/ISO11/stdlib.h/conform
FAIL: conform/ISO11/stdlib.h/linknamespace
FAIL: conform/ISO11/wchar.h/conform
FAIL: conform/ISO99/setjmp.h/conform
FAIL: conform/ISO99/stdio.h/conform
FAIL: conform/ISO99/stdlib.h/conform
FAIL: conform/ISO99/stdlib.h/linknamespace
FAIL: conform/ISO99/wchar.h/conform
FAIL: conform/POSIX/stdio.h/conform
FAIL: conform/POSIX/stdlib.h/conform
FAIL: conform/POSIX/stdlib.h/linknamespace
FAIL: conform/POSIX2008/arpa/inet.h/conform
FAIL: conform/POSIX2008/fcntl.h/conform
FAIL: conform/POSIX2008/mqueue.h/conform
FAIL: conform/POSIX2008/netdb.h/conform
FAIL: conform/POSIX2008/netinet/in.h/conform
FAIL: conform/POSIX2008/stdio.h/conform
FAIL: conform/POSIX2008/stdlib.h/conform
FAIL: conform/POSIX2008/stdlib.h/linknamespace
FAIL: conform/POSIX2008/sys/socket.h/conform
FAIL: conform/POSIX2008/unistd.h/linknamespace
FAIL: conform/POSIX2008/wchar.h/conform
FAIL: conform/UNIX98/arpa/inet.h/conform
XPASS: conform/UNIX98/ndbm.h/linknamespace
FAIL: conform/UNIX98/netdb.h/conform
FAIL: conform/UNIX98/netinet/in.h/conform
FAIL: conform/UNIX98/stdio.h/conform
FAIL: conform/UNIX98/stdlib.h/conform
FAIL: conform/UNIX98/sys/socket.h/conform
FAIL: conform/UNIX98/unistd.h/conform
FAIL: conform/UNIX98/unistd.h/linknamespace
FAIL: conform/UNIX98/wchar.h/conform
FAIL: conform/XOPEN2K/arpa/inet.h/conform
FAIL: conform/XOPEN2K/fcntl.h/conform
FAIL: conform/XOPEN2K/mqueue.h/conform
XPASS: conform/XOPEN2K/ndbm.h/linknamespace
FAIL: conform/XOPEN2K/netdb.h/conform
FAIL: conform/XOPEN2K/netinet/in.h/conform
FAIL: conform/XOPEN2K/stdio.h/conform
FAIL: conform/XOPEN2K/stdlib.h/conform
FAIL: conform/XOPEN2K/sys/socket.h/conform
FAIL: conform/XOPEN2K/syslog.h/conform
FAIL: conform/XOPEN2K/unistd.h/conform
FAIL: conform/XOPEN2K/unistd.h/linknamespace
FAIL: conform/XOPEN2K/wchar.h/conform
FAIL: conform/XOPEN2K8/arpa/inet.h/conform
FAIL: conform/XOPEN2K8/fcntl.h/conform
FAIL: conform/XOPEN2K8/mqueue.h/conform
XPASS: conform/XOPEN2K8/ndbm.h/linknamespace
FAIL: conform/XOPEN2K8/netdb.h/conform
FAIL: conform/XOPEN2K8/netinet/in.h/conform
FAIL: conform/XOPEN2K8/stdio.h/conform
FAIL: conform/XOPEN2K8/stdlib.h/conform
FAIL: conform/XOPEN2K8/sys/socket.h/conform
FAIL: conform/XOPEN2K8/syslog.h/conform
FAIL: conform/XOPEN2K8/unistd.h/conform
FAIL: conform/XOPEN2K8/unistd.h/linknamespace
FAIL: conform/XOPEN2K8/wchar.h/conform
FAIL: conform/XPG4/stdio.h/conform
FAIL: conform/XPG4/stdlib.h/conform
FAIL: conform/XPG4/stdlib.h/linknamespace
FAIL: conform/XPG4/unistd.h/conform
FAIL: conform/XPG42/arpa/inet.h/conform
XPASS: conform/XPG42/ndbm.h/linknamespace
FAIL: conform/XPG42/netdb.h/conform
FAIL: conform/XPG42/netinet/in.h/conform
FAIL: conform/XPG42/stdio.h/conform
FAIL: conform/XPG42/stdlib.h/conform
FAIL: conform/XPG42/sys/socket.h/conform
FAIL: conform/XPG42/unistd.h/conform

You will need to check on the *.out generated file to see which symbols is
being triggering this.  It might a wrong toolchain build pulling some 
unexpected symbol.


FAIL: elf/circleload1
FAIL: elf/constload1
FAIL: elf/dblload
FAIL: elf/dblunload
FAIL: elf/lateglobal
FAIL: elf/reldep6
FAIL: elf/resolvfail
FAIL: elf/tst-bz15311
FAIL: elf/tst-global1
FAIL: elf/tst-tls20

As before, it will require to check the *.out file.

FAIL: localedata/sort-test
FAIL: localedata/tst-mbswcs6
FAIL: localedata/tst-numeric
FAIL: localedata/tst-setlocale2
FAIL: localedata/tst-setlocale3
FAIL: localedata/tst-sscanf
FAIL: localedata/tst-strfmon1

I am not sure what is state of locale tests for cross-tests. It might
require some fixes here.

FAIL: malloc/tst-malloc-thread-fail
FAIL: malloc/tst-malloc-thread-fail-malloc-check
FAIL: malloc/tst-malloc-thread-fail-malloc-hugetlb1
FAIL: malloc/tst-malloc-thread-fail-malloc-hugetlb2
FAIL: malloc/tst-malloc_info
FAIL: malloc/tst-malloc_info-malloc-check
FAIL: malloc/tst-malloc_info-malloc-hugetlb1
FAIL: malloc/tst-malloc_info-malloc-hugetlb2

I think these might be due default timeout if too low depending of the
target machine.  Check the *.out file and you can use a larger default
timeout by exporting TIMEOUTFACTOR=N environment variable.

Ideally we want to setup timeout for tests that are not susceptible
to machine speed (although depending of the system load it might still
trigger timeouts).

FAIL: posix/bug-regex18
FAIL: posix/bug-regex35
FAIL: posix/globtest
FAIL: posix/tst-fnmatch4
FAIL: posix/tst-rxspencer
FAIL: stdlib/test-bz22786
FAIL: stdlib/tst-strfmon_l
FAIL: stdlib/tst-strtod-nan-locale
FAIL: stdlib/tst-strtod4
FAIL: stdlib/tst-strtod5
FAIL: stdlib/tst-strtod5i
FAIL: stdlib/tst-strtol-locale
FAIL: string/bug-strcoll2

FAIL: time/tst-strptime
FAIL: timezone/tst-tzset
FAIL: wcsmbs/tst-btowc
FAIL: wcsmbs/tst-wcstod-nan-locale
FAIL: wcsmbs/tst-wcstol-locale

I think these are related to locales missing, although I am not sure
why exactly.

FAIL: string/test-memcpy-large

This might also be a timeout issue.

FAIL: string/test-strcasecmp
FAIL: string/test-strncasecmp
FAIL: string/tst-strerror
FAIL: string/tst-strsignal

These use containers, you will need to check the *.out file to see
exactly what is happening.