failures on Raspberry Pi

failures on Raspberry Pi

[Paul Zimmermann]

       Hi,

I get 8 test failures on a Raspberry Pi (ARMv7 Processor rev 5 (v7l)) with
commit 934c48a and gcc 10.2.1:

Summary of test results:
      8 FAIL
   3930 PASS
     18 UNSUPPORTED
     17 XFAIL
      2 XPASS

FAIL: elf/tst-glibc-hwcaps-prepend-cache
tst-glibc-hwcaps-prepend-cache.c:109: numeric comparison failure
   left: 2 (0x2); from: marker1 ()
  right: 3 (0x3); from: 3
error: tst-glibc-hwcaps-prepend-cache.c:118: not true: dlopen (SONAME, RTLD_NOW) == NULL
error: 2 test failures
running post-clean rsync

FAIL: locale/tst-localedef-path-norm
info: Directory '/usr/lib/locale/en_US1.utf8' exists.
Timed out: killed the child process
Termination time: 2021-01-29T12:51:15.067421847
Last write to standard output: 2021-01-29T12:51:10.320779487
running post-clean rsync

FAIL: localedata/sort-test
cmn_TW.UTF-8 xfrm-test FAIL

FAIL: misc/tst-sysvshm-linux
../sysdeps/unix/sysv/linux/tst-sysvshm-linux.c:155: numeric comparison failure
   left: 4278190079 (0xfeffffff); from: ipcinfo.shmmax
  right: 2147483647 (0x7fffffff); from: tipcinfo.shmmax
error: 1 test failures

FAIL: nss/tst-nss-files-hosts-getent
Timed out: killed the child process
Termination time: 2021-01-29T16:20:09.398286002
Last write to standard output: 2021-01-29T16:19:49.383886436

FAIL: nss/tst-nss-files-hosts-multi
Timed out: killed the child process
Termination time: 2021-01-29T16:19:47.876326179
Last write to standard output: 2021-01-29T16:19:07.863774643

FAIL: stdio-common/tst-vfprintf-width-prec
Timed out: killed the child process
Termination time: 2021-01-29T14:17:07.870905125
Last write to standard output: 2021-01-29T14:16:47.850422358

FAIL: stdio-common/tst-vfprintf-width-prec-mem
Memory not freed:
-----------------
   Address     Size     Caller
0x70a5f008 0x31fff9c  at 0x76e814cf

I can provide more details if needed. For the tests with "timed out", is there
a way to increase the default timeout threshold?

Paul

Re: failures on Raspberry Pi

[Florian Weimer]

* Paul Zimmermann:

> I can provide more details if needed. For the tests with "timed out", is there
> a way to increase the default timeout threshold?

There is TIMEOUTFACTOR for that.

Re: failures on Raspberry Pi

[Paul Zimmermann]

       Florian,

> > I can provide more details if needed. For the tests with "timed out", is there
> > a way to increase the default timeout threshold?
> 
> There is TIMEOUTFACTOR for that.

thank you, with TIMEOUTFACTOR=1000 I still get 3 failures:

FAIL: elf/tst-glibc-hwcaps-prepend-cache
FAIL: localedata/sort-test
FAIL: misc/tst-sysvshm-linux

$ cat elf/tst-glibc-hwcaps-prepend-cache.out
tst-glibc-hwcaps-prepend-cache.c:109: numeric comparison failure
   left: 2 (0x2); from: marker1 ()
  right: 3 (0x3); from: 3
error: tst-glibc-hwcaps-prepend-cache.c:118: not true: dlopen (SONAME, RTLD_NOW) == NULL
error: 2 test failures
running post-clean rsync

$ grep FAIL localedata/sort-test.out
cmn_TW.UTF-8 xfrm-test FAIL

$ cat misc/tst-sysvshm-linux.out
../sysdeps/unix/sysv/linux/tst-sysvshm-linux.c:155: numeric comparison failure
   left: 4278190079 (0xfeffffff); from: ipcinfo.shmmax
  right: 2147483647 (0x7fffffff); from: tipcinfo.shmmax
error: 1 test failures

Are those expected?

Paul

Re: failures on Raspberry Pi

[Adhemerval Zanella]

On 01/02/2021 18:09, Paul Zimmermann wrote:
>        Florian,
> 
>>> I can provide more details if needed. For the tests with "timed out", is there
>>> a way to increase the default timeout threshold?
>>
>> There is TIMEOUTFACTOR for that.
> 
> thank you, with TIMEOUTFACTOR=1000 I still get 3 failures:
> 
> FAIL: elf/tst-glibc-hwcaps-prepend-cache
> FAIL: localedata/sort-test
> FAIL: misc/tst-sysvshm-linux
> 
> $ cat elf/tst-glibc-hwcaps-prepend-cache.out
> tst-glibc-hwcaps-prepend-cache.c:109: numeric comparison failure
>    left: 2 (0x2); from: marker1 ()
>   right: 3 (0x3); from: 3
> error: tst-glibc-hwcaps-prepend-cache.c:118: not true: dlopen (SONAME, RTLD_NOW) == NULL
> error: 2 test failures
> running post-clean rsync

I think this is the missing ld.so.cache for cross-compiling which
should be fixed by a50a080f6b0f958eb30d7520212f25f946ed7f8f.

> 
> $ grep FAIL localedata/sort-test.out
> cmn_TW.UTF-8 xfrm-test FAIL

I am not sure about this one.

> 
> $ cat misc/tst-sysvshm-linux.out
> ../sysdeps/unix/sysv/linux/tst-sysvshm-linux.c:155: numeric comparison failure
>    left: 4278190079 (0xfeffffff); from: ipcinfo.shmmax
>   right: 2147483647 (0x7fffffff); from: tipcinfo.shmmax
> error: 1 test failures

I noticed it on some 32-bit architectures running on 32-bit kernels.  The issue 
is shmctl (IPC_INFO) clamps shmmax on for:

  1. Compat symbols with IPC_64, i.e, 32-bit binaries running on 64-bit kernels.

1246 static int copy_compat_shminfo_to_user(void __user *buf, struct shminfo64 *in,
1247                                         int version)
1248 {
1249         if (in->shmmax > INT_MAX)
1250                 in->shmmax = INT_MAX;

  2. Default symbol with IPC_OLD and glibc always use IPC_64 now. It means that
     32-bit binaries running on 32-bit kernels will not see shmmax being clamped
     anymore.

 801 static inline unsigned long copy_shminfo_to_user(void __user *buf, struct shminfo64 *in, int version)
 802 {
 803         switch (version) {
 804         case IPC_64:
 805                 return copy_to_user(buf, in, sizeof(*in));
 806         case IPC_OLD:
 807             {
 808                 struct shminfo out;
 809 
 810                 if (in->shmmax > INT_MAX)
 811                         out.shmmax = INT_MAX;
 812                 else
 813                         out.shmmax = (int)in->shmmax;

It means that we will need to check if whether the kernel will use the compat
symbols to properly handle on tst-sysvshm-linux.  I am more inclined in just
remove the shmmax tests and add a comment why it is complicated to do a proper
check.

> 
> Are those expected?
> 
> Paul
> 

Re: failures on Raspberry Pi

[Joseph Myers]

On Mon, 1 Feb 2021, Paul Zimmermann wrote:

> > > I can provide more details if needed. For the tests with "timed out", is there
> > > a way to increase the default timeout threshold?
> > 
> > There is TIMEOUTFACTOR for that.
> 
> thank you, with TIMEOUTFACTOR=1000 I still get 3 failures:

Note that we also sometimes increase TIMEOUT in individual affected tests, 
to reduce the need for setting TIMEOUTFACTOR - so finding settings that 
work reliably for the tests that timed out without setting TIMEOUTFACTOR, 
then updating those tests accordingly, might be a good idea.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: failures on Raspberry Pi

[Carlos O'Donell]

On 1/31/21 2:33 AM, Paul Zimmermann wrote:
> FAIL: localedata/sort-test
> cmn_TW.UTF-8 xfrm-test FAIL

That's an odd failure and unexpected.

It's not always easy to determine the exact cause of the problem.

-- 
Cheers,
Carlos.

Re: failures on Raspberry Pi

[Florian Weimer]

* Carlos O'Donell via Libc-alpha:

> On 1/31/21 2:33 AM, Paul Zimmermann wrote:
>> FAIL: localedata/sort-test
>> cmn_TW.UTF-8 xfrm-test FAIL
>
> That's an odd failure and unexpected.
>
> It's not always easy to determine the exact cause of the problem.

Maybe the locale files got corrupted due to an out-of-memory condition?
cmn_TW.UTF-8 seems to be the locale that requires the most memory (see
the recent discussion).

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: failures on Raspberry Pi

[H.J. Lu]

On Tue, Feb 2, 2021 at 11:47 AM Florian Weimer via Libc-alpha
<libc-alpha@sourceware.org> wrote:
>
> * Carlos O'Donell via Libc-alpha:
>
> > On 1/31/21 2:33 AM, Paul Zimmermann wrote:
> >> FAIL: localedata/sort-test
> >> cmn_TW.UTF-8 xfrm-test FAIL
> >
> > That's an odd failure and unexpected.
> >
> > It's not always easy to determine the exact cause of the problem.
>
> Maybe the locale files got corrupted due to an out-of-memory condition?
> cmn_TW.UTF-8 seems to be the locale that requires the most memory (see
> the recent discussion).
>

This may be related to

https://sourceware.org/bugzilla/show_bug.cgi?id=27320

-- 
H.J.

Re: failures on Raspberry Pi

[Paul Zimmermann]

       Hi,

> Date: Tue, 2 Feb 2021 12:32:20 -0800
> From: "H.J. Lu via Libc-alpha" <libc-alpha@sourceware.org>
> Cc: Carlos O'Donell via Libc-alpha <libc-alpha@sourceware.org>
> 
> On Tue, Feb 2, 2021 at 11:47 AM Florian Weimer via Libc-alpha
> <libc-alpha@sourceware.org> wrote:
> >
> > * Carlos O'Donell via Libc-alpha:
> >
> > > On 1/31/21 2:33 AM, Paul Zimmermann wrote:
> > >> FAIL: localedata/sort-test
> > >> cmn_TW.UTF-8 xfrm-test FAIL
> > >
> > > That's an odd failure and unexpected.
> > >
> > > It's not always easy to determine the exact cause of the problem.
> >
> > Maybe the locale files got corrupted due to an out-of-memory condition?
> > cmn_TW.UTF-8 seems to be the locale that requires the most memory (see
> > the recent discussion).
> >
> 
> This may be related to
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=27320

I'm not sure: if I apply the same patch (adding "hlt"), I get:

zimmerma@calva:/tmp/glibc/build$ make subdirs=string -j1 check
...
UNSUPPORTED: string/test-bcopy
UNSUPPORTED: string/test-memmove
UNSUPPORTED: string/tst-memmove-overflow
UNSUPPORTED: string/tst-strerror
UNSUPPORTED: string/tst-strsignal
Summary of test results:
     72 PASS
      5 UNSUPPORTED

Paul

Re: failures on Raspberry Pi

[Paul Zimmermann]

       Dear Adhemerval,

> > $ cat elf/tst-glibc-hwcaps-prepend-cache.out
> > tst-glibc-hwcaps-prepend-cache.c:109: numeric comparison failure
> >    left: 2 (0x2); from: marker1 ()
> >   right: 3 (0x3); from: 3
> > error: tst-glibc-hwcaps-prepend-cache.c:118: not true: dlopen (SONAME, RTLD_NOW) == NULL
> > error: 2 test failures
> > running post-clean rsync
> 
> I think this is the missing ld.so.cache for cross-compiling which
> should be fixed by a50a080f6b0f958eb30d7520212f25f946ed7f8f.

indeed, with revision 9132010 I get:

$ make subdirs=elf -j1 check
...
    261 PASS
      1 UNSUPPORTED
      2 XPASS

Paul

Re: failures on Raspberry Pi

[Paul Zimmermann]

       Dear Florian,

> > On 1/31/21 2:33 AM, Paul Zimmermann wrote:
> >> FAIL: localedata/sort-test
> >> cmn_TW.UTF-8 xfrm-test FAIL
> >
> > That's an odd failure and unexpected.
> >
> > It's not always easy to determine the exact cause of the problem.
> 
> Maybe the locale files got corrupted due to an out-of-memory condition?
> cmn_TW.UTF-8 seems to be the locale that requires the most memory (see
> the recent discussion).

this is most likely. During localedata/sort-test top shows RES going up to
700MB. Can't we reduce the memory needed for that test?

Paul

Re: failures on Raspberry Pi

[Paul Zimmermann]

       Dear Adhemerval,

> > $ cat misc/tst-sysvshm-linux.out
> > ../sysdeps/unix/sysv/linux/tst-sysvshm-linux.c:155: numeric comparison failure
> >    left: 4278190079 (0xfeffffff); from: ipcinfo.shmmax
> >   right: 2147483647 (0x7fffffff); from: tipcinfo.shmmax
> > error: 1 test failures
> 
> I noticed it on some 32-bit architectures running on 32-bit kernels.  The issue 
> is shmctl (IPC_INFO) clamps shmmax on for [...]

all misc tests pass after commit 9132010:

$ make subdirs=misc -j1 check
...
Summary of test results:
     68 PASS
      1 UNSUPPORTED

In summary, it should remain only one failure with TIMEOUTFACTOR=1000
(cmn_TW.UTF-8 xfrm-test from localedata/sort-test.out, using too much memory).

Paul