glibc test procedure needed

glibc test procedure needed

[sundeep.kokkonda]

Hello,

 

I need some info on glibc testing.

 

In the Yocto project we're updating glibc frequently and so we need to
ensure the updated glibc is not introduced any issues, by performing the
testing.

Can you let me know how can I test glibc? Is there any standardized test
procedure available to do full test/regression test?

 

 

 

Thanks,

Sundeep K.

Re: glibc test procedure needed

[Adhemerval Zanella]

> On 5 Jul 2022, at 03:09, sundeep.kokkonda--- via Libc-alpha <libc-alpha@sourceware.org> wrote:
> 
> Hello,
> 
> 
> 
> I need some info on glibc testing.
> 
> 
> 
> In the Yocto project we're updating glibc frequently and so we need to
> ensure the updated glibc is not introduced any issues, by performing the
> testing.
> 
> Can you let me know how can I test glibc? Is there any standardized test
> procedure available to do full test/regression test?
> 

The usual ‘make check’ should cover all the required testing, you might
need to use ‘run-built-tests=yes’ additional directive if you are
cross-compiling to force the make to issues the built tests (for instance
on architecture that support multiple ABI, like x86, or to force the
use of some emulation like qemu-user).

We are still fixing up some clunky tests and there are some broken 
environment (for instance broken syscall filtering) where some tests
might advertise a failure.

RE: glibc test procedure needed

[sundeep.kokkonda]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

> -----Original Message-----
> From: sundeep.kokkonda@gmail.com <sundeep.kokkonda@gmail.com>
> Sent: 08 July 2022 10:20
> To: 'Adhemerval Zanella' <adhemerval.zanella@linaro.org>
> Cc: 'Libc-help' <libc-help@sourceware.org>
> Subject: RE: glibc test procedure needed
> 
> 
> 
> > -----Original Message-----
> > From: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> > Sent: 05 July 2022 18:13
> > To: sundeep.kokkonda@gmail.com
> > Cc: Libc-help <libc-help@sourceware.org>
> > Subject: Re: glibc test procedure needed
> >
> >
> >
> > > On 5 Jul 2022, at 03:09, sundeep.kokkonda--- via Libc-alpha <libc-
> > alpha@sourceware.org> wrote:
> > >
> > > Hello,
> > >
> > >
> > >
> > > I need some info on glibc testing.
> > >
> > >
> > >
> > > In the Yocto project we're updating glibc frequently and so we need
> > > to ensure the updated glibc is not introduced any issues, by
> > > performing the testing.
> > >
> > > Can you let me know how can I test glibc? Is there any standardized
> > > test procedure available to do full test/regression test?
> > >
> >
> > The usual ‘make check’ should cover all the required testing, you
> > might need to use ‘run-built-tests=yes’ additional directive if you
> > are cross-compiling to force the make to issues the built tests (for
> > instance on architecture that support multiple ABI, like x86, or to force the use
> of some emulation like qemu-user).
> >
> > We are still fixing up some clunky tests and there are some broken
> > environment (for instance broken syscall filtering) where some tests might
> advertise a failure.
> 
> Hello,
> 
> I tried with 'Testing with a cross-compiler' method and test is aborted when the
> flag '-Wformat-security' is enabled. When this flag is removed the test is
> executed successfully. Is it a known issue? (The error log make-check.out is
> attached for reference) Also, can anyone share the list of known test failures (if
> any).
> 

Hello,

Can I get some info regarding my query.

> 
> Thanks,
> Sundeep K.


[-- Attachment #2: make-check.zip --]
[-- Type: application/x-zip-compressed, Size: 391875 bytes --]

Re: glibc test procedure needed

[Adhemerval Zanella]

> On 8 Jul 2022, at 01:49, sundeep.kokkonda@gmail.com wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>> Sent: 05 July 2022 18:13
>> To: sundeep.kokkonda@gmail.com
>> Cc: Libc-help <libc-help@sourceware.org>
>> Subject: Re: glibc test procedure needed
>> 
>> 
>> 
>>> On 5 Jul 2022, at 03:09, sundeep.kokkonda--- via Libc-alpha <libc-
>> alpha@sourceware.org> wrote:
>>> 
>>> Hello,
>>> 
>>> 
>>> 
>>> I need some info on glibc testing.
>>> 
>>> 
>>> 
>>> In the Yocto project we're updating glibc frequently and so we need to
>>> ensure the updated glibc is not introduced any issues, by performing
>>> the testing.
>>> 
>>> Can you let me know how can I test glibc? Is there any standardized
>>> test procedure available to do full test/regression test?
>>> 
>> 
>> The usual ‘make check’ should cover all the required testing, you might need to
>> use ‘run-built-tests=yes’ additional directive if you are cross-compiling to force
>> the make to issues the built tests (for instance on architecture that support
>> multiple ABI, like x86, or to force the use of some emulation like qemu-user).
>> 
>> We are still fixing up some clunky tests and there are some broken environment
>> (for instance broken syscall filtering) where some tests might advertise a failure.
> 
> Hello,
> 
> I tried with 'Testing with a cross-compiler' method and test is aborted when the flag '-Wformat-security' is enabled. When this flag is removed the test is executed successfully. Is it a known issue? (The error log make-check.out is attached for reference)
> Also, can anyone share the list of known test failures (if any).
> 
> 
> Thanks,
> Sundeep K.
> 
> <make-check.out>


Since glibc enables -Werror as default, not all compiler flags are supported
without triggering warning that might abort the build.  Ideally for each new
warning flag you will first build with —disable-error and check each error
to possible either check if this is a compiler issue or if it requires patch
glibc.

For the ‘make check’ output you can see why might be happening by checking
each file.out file:

UNSUPPORTED: assert/tst-assert-c++
UNSUPPORTED: assert/tst-assert-g++

It means that you do not configure with a C++ compiler (it is required only
for testing).

FAIL: catgets/test1.cat

I am not sure about this one, but it seems that is a locale not being present
in the testing.

FAIL: conform/ISO/setjmp.h/conform
FAIL: conform/ISO/stdio.h/conform
FAIL: conform/ISO/stdlib.h/conform
FAIL: conform/ISO/stdlib.h/linknamespace
FAIL: conform/ISO11/setjmp.h/conform
FAIL: conform/ISO11/stdio.h/conform
FAIL: conform/ISO11/stdlib.h/conform
FAIL: conform/ISO11/stdlib.h/linknamespace
FAIL: conform/ISO11/wchar.h/conform
FAIL: conform/ISO99/setjmp.h/conform
FAIL: conform/ISO99/stdio.h/conform
FAIL: conform/ISO99/stdlib.h/conform
FAIL: conform/ISO99/stdlib.h/linknamespace
FAIL: conform/ISO99/wchar.h/conform
FAIL: conform/POSIX/stdio.h/conform
FAIL: conform/POSIX/stdlib.h/conform
FAIL: conform/POSIX/stdlib.h/linknamespace
FAIL: conform/POSIX2008/arpa/inet.h/conform
FAIL: conform/POSIX2008/fcntl.h/conform
FAIL: conform/POSIX2008/mqueue.h/conform
FAIL: conform/POSIX2008/netdb.h/conform
FAIL: conform/POSIX2008/netinet/in.h/conform
FAIL: conform/POSIX2008/stdio.h/conform
FAIL: conform/POSIX2008/stdlib.h/conform
FAIL: conform/POSIX2008/stdlib.h/linknamespace
FAIL: conform/POSIX2008/sys/socket.h/conform
FAIL: conform/POSIX2008/unistd.h/linknamespace
FAIL: conform/POSIX2008/wchar.h/conform
FAIL: conform/UNIX98/arpa/inet.h/conform
XPASS: conform/UNIX98/ndbm.h/linknamespace
FAIL: conform/UNIX98/netdb.h/conform
FAIL: conform/UNIX98/netinet/in.h/conform
FAIL: conform/UNIX98/stdio.h/conform
FAIL: conform/UNIX98/stdlib.h/conform
FAIL: conform/UNIX98/sys/socket.h/conform
FAIL: conform/UNIX98/unistd.h/conform
FAIL: conform/UNIX98/unistd.h/linknamespace
FAIL: conform/UNIX98/wchar.h/conform
FAIL: conform/XOPEN2K/arpa/inet.h/conform
FAIL: conform/XOPEN2K/fcntl.h/conform
FAIL: conform/XOPEN2K/mqueue.h/conform
XPASS: conform/XOPEN2K/ndbm.h/linknamespace
FAIL: conform/XOPEN2K/netdb.h/conform
FAIL: conform/XOPEN2K/netinet/in.h/conform
FAIL: conform/XOPEN2K/stdio.h/conform
FAIL: conform/XOPEN2K/stdlib.h/conform
FAIL: conform/XOPEN2K/sys/socket.h/conform
FAIL: conform/XOPEN2K/syslog.h/conform
FAIL: conform/XOPEN2K/unistd.h/conform
FAIL: conform/XOPEN2K/unistd.h/linknamespace
FAIL: conform/XOPEN2K/wchar.h/conform
FAIL: conform/XOPEN2K8/arpa/inet.h/conform
FAIL: conform/XOPEN2K8/fcntl.h/conform
FAIL: conform/XOPEN2K8/mqueue.h/conform
XPASS: conform/XOPEN2K8/ndbm.h/linknamespace
FAIL: conform/XOPEN2K8/netdb.h/conform
FAIL: conform/XOPEN2K8/netinet/in.h/conform
FAIL: conform/XOPEN2K8/stdio.h/conform
FAIL: conform/XOPEN2K8/stdlib.h/conform
FAIL: conform/XOPEN2K8/sys/socket.h/conform
FAIL: conform/XOPEN2K8/syslog.h/conform
FAIL: conform/XOPEN2K8/unistd.h/conform
FAIL: conform/XOPEN2K8/unistd.h/linknamespace
FAIL: conform/XOPEN2K8/wchar.h/conform
FAIL: conform/XPG4/stdio.h/conform
FAIL: conform/XPG4/stdlib.h/conform
FAIL: conform/XPG4/stdlib.h/linknamespace
FAIL: conform/XPG4/unistd.h/conform
FAIL: conform/XPG42/arpa/inet.h/conform
XPASS: conform/XPG42/ndbm.h/linknamespace
FAIL: conform/XPG42/netdb.h/conform
FAIL: conform/XPG42/netinet/in.h/conform
FAIL: conform/XPG42/stdio.h/conform
FAIL: conform/XPG42/stdlib.h/conform
FAIL: conform/XPG42/sys/socket.h/conform
FAIL: conform/XPG42/unistd.h/conform

You will need to check on the *.out generated file to see which symbols is
being triggering this.  It might a wrong toolchain build pulling some 
unexpected symbol.


FAIL: elf/circleload1
FAIL: elf/constload1
FAIL: elf/dblload
FAIL: elf/dblunload
FAIL: elf/lateglobal
FAIL: elf/reldep6
FAIL: elf/resolvfail
FAIL: elf/tst-bz15311
FAIL: elf/tst-global1
FAIL: elf/tst-tls20

As before, it will require to check the *.out file.

FAIL: localedata/sort-test
FAIL: localedata/tst-mbswcs6
FAIL: localedata/tst-numeric
FAIL: localedata/tst-setlocale2
FAIL: localedata/tst-setlocale3
FAIL: localedata/tst-sscanf
FAIL: localedata/tst-strfmon1

I am not sure what is state of locale tests for cross-tests. It might
require some fixes here.

FAIL: malloc/tst-malloc-thread-fail
FAIL: malloc/tst-malloc-thread-fail-malloc-check
FAIL: malloc/tst-malloc-thread-fail-malloc-hugetlb1
FAIL: malloc/tst-malloc-thread-fail-malloc-hugetlb2
FAIL: malloc/tst-malloc_info
FAIL: malloc/tst-malloc_info-malloc-check
FAIL: malloc/tst-malloc_info-malloc-hugetlb1
FAIL: malloc/tst-malloc_info-malloc-hugetlb2

I think these might be due default timeout if too low depending of the
target machine.  Check the *.out file and you can use a larger default
timeout by exporting TIMEOUTFACTOR=N environment variable.

Ideally we want to setup timeout for tests that are not susceptible
to machine speed (although depending of the system load it might still
trigger timeouts).

FAIL: posix/bug-regex18
FAIL: posix/bug-regex35
FAIL: posix/globtest
FAIL: posix/tst-fnmatch4
FAIL: posix/tst-rxspencer
FAIL: stdlib/test-bz22786
FAIL: stdlib/tst-strfmon_l
FAIL: stdlib/tst-strtod-nan-locale
FAIL: stdlib/tst-strtod4
FAIL: stdlib/tst-strtod5
FAIL: stdlib/tst-strtod5i
FAIL: stdlib/tst-strtol-locale
FAIL: string/bug-strcoll2

FAIL: time/tst-strptime
FAIL: timezone/tst-tzset
FAIL: wcsmbs/tst-btowc
FAIL: wcsmbs/tst-wcstod-nan-locale
FAIL: wcsmbs/tst-wcstol-locale

I think these are related to locales missing, although I am not sure
why exactly.

FAIL: string/test-memcpy-large

This might also be a timeout issue.

FAIL: string/test-strcasecmp
FAIL: string/test-strncasecmp
FAIL: string/tst-strerror
FAIL: string/tst-strsignal

These use containers, you will need to check the *.out file to see
exactly what is happening.