state of the ports

state of the ports

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]

So... after all the testing gone into the release I thought I'd summarize a bit
for the list here - by sorting the glibc ports into "excellent", "good", "medium",
"bad", and "nobody even tried".

Feel free to correct my assessment... Most of the underlying info is on the
2.39 release wiki page.

Cheers,
Andreas


1. EXCELLENT 
============

aarch64
  - tests pass nearly everywhere
  - on raspi, likely timeouts

arm-linux-gnueabi (soft-float) 
arm-linux-gnueabihf (hard-float) 
  - one mystery test failure (elf/tst-align3)

loongarch64 
  - tests pass

powerpc64
powerpc64le
  - tests pass

s390x
  - tests pass

x86-64
  - tests pass

x86-64 (x32)
  - one expected failure, tests pass otherwise


2. GOOD
============

powerpc64
ppc 32bit hf
  - one failure that indicates something broken (nptl/tst-thread-exit-clobber)

riscv64
  - ~5 failures, but there may be timeouts among them

s390 (31bit)
  - two failures due to gcc breakage (gmon/tst-gmon-pie*)

sparc
  - ~5 failures (!)

x86 (32bit)
  - in general tests pass
  - there may be breakage on old hw (pentium2)


3. MEDIUM
============

ARC
 - 11 failures

hppa2.0
 - 12 failures (on ~20 year old hw! that's actually good)


4. BAD
============

Debian
 - OK not really a port :) but it seems there are some machines where
   broken Debian installs lead to massive amounts of FAIL

mips (o32)
 - actually broken right now, it seems, the tests came too late

mips (n64)
 - mostly math failures

riscv32
 - 37 failures across the board


99. NOBODY TRIED
=================

alpha

c-sky

hppa1.1

m68k

microblaze

mips64 (N32)

niosii

openrisc

superh



-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: state of the ports

[Xi Ruoyao]

On Fri, 2024-02-02 at 03:35 +0100, Andreas K. Huettel wrote:
> mips (n64)
>  - mostly math failures

They are some sort of 2 ulp vs expected 1 ulp things.  I'm not yet sure
if they are hardware issues or Glibc bugs.

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: state of the ports

[Xi Ruoyao]

On Fri, 2024-02-02 at 15:51 +0800, Xi Ruoyao wrote:
> On Fri, 2024-02-02 at 03:35 +0100, Andreas K. Huettel wrote:
> > mips (n64)
> >  - mostly math failures
> 
> They are some sort of 2 ulp vs expected 1 ulp things.  I'm not yet sure
> if they are hardware issues or Glibc bugs.

Something like:

testing double (without inline functions)
Failure: Test: j0 (0x4.130e7p+4)
Result:
 is:         -1.0474906594151864e-04  -0x1.b759740016860p-14
 should be:  -1.0474906594151868e-04  -0x1.b759740016863p-14
 difference:  4.0657581468206416e-20   0x1.8000000000000p-65
 ulp       :  3.0000
 max.ulp   :  2.0000
Maximal error of `j0'
 is      : 3 ulp
 accepted: 2 ulp
Failure: Test: j0_downward (0xa.5b833p+4)
Result:
 is:          1.4785400727226293e-05   0x1.f01da00ab51d1p-17
 should be:   1.4785400727226283e-05   0x1.f01da00ab51cbp-17
 difference:  1.0164395367051604e-20   0x1.8000000000000p-67
 ulp       :  6.0000
 max.ulp   :  5.0000
Maximal error of `j0_downward'
 is      : 6 ulp
 accepted: 5 ulp
Failure: Test: j0_towardzero (0xa.5b833p+4)
Result:
 is:          1.4785400727226271e-05   0x1.f01da00ab51c4p-17
 should be:   1.4785400727226283e-05   0x1.f01da00ab51cbp-17
 difference:  1.1858461261560204e-20   0x1.c000000000000p-67
 ulp       :  7.0000
 max.ulp   :  6.0000
Maximal error of `j0_towardzero'
 is      : 7 ulp
 accepted: 6 ulp

Test suite completed:
  212 test cases plus 208 tests for exception flags and
    208 tests for errno executed.
  6 errors occurred.

I'm really not an expert in numerical analysis.  Any pointer to debug
such a thing further?

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: state of the ports

[Florian Weimer]

* Xi Ruoyao:

> On Fri, 2024-02-02 at 15:51 +0800, Xi Ruoyao wrote:
>> On Fri, 2024-02-02 at 03:35 +0100, Andreas K. Huettel wrote:
>> > mips (n64)
>> >  - mostly math failures
>> 
>> They are some sort of 2 ulp vs expected 1 ulp things.  I'm not yet sure
>> if they are hardware issues or Glibc bugs.
>
> Something like:
>
> testing double (without inline functions)
> Failure: Test: j0 (0x4.130e7p+4)
> Result:
>  is:         -1.0474906594151864e-04  -0x1.b759740016860p-14
>  should be:  -1.0474906594151868e-04  -0x1.b759740016863p-14
>  difference:  4.0657581468206416e-20   0x1.8000000000000p-65
>  ulp       :  3.0000
>  max.ulp   :  2.0000
> Maximal error of `j0'
>  is      : 3 ulp
>  accepted: 2 ulp
> Failure: Test: j0_downward (0xa.5b833p+4)
> Result:
>  is:          1.4785400727226293e-05   0x1.f01da00ab51d1p-17
>  should be:   1.4785400727226283e-05   0x1.f01da00ab51cbp-17
>  difference:  1.0164395367051604e-20   0x1.8000000000000p-67
>  ulp       :  6.0000
>  max.ulp   :  5.0000
> Maximal error of `j0_downward'
>  is      : 6 ulp
>  accepted: 5 ulp
> Failure: Test: j0_towardzero (0xa.5b833p+4)
> Result:
>  is:          1.4785400727226271e-05   0x1.f01da00ab51c4p-17
>  should be:   1.4785400727226283e-05   0x1.f01da00ab51cbp-17
>  difference:  1.1858461261560204e-20   0x1.c000000000000p-67
>  ulp       :  7.0000
>  max.ulp   :  6.0000
> Maximal error of `j0_towardzero'
>  is      : 7 ulp
>  accepted: 6 ulp
>
> Test suite completed:
>   212 test cases plus 208 tests for exception flags and
>     208 tests for errno executed.
>   6 errors occurred.
>
> I'm really not an expert in numerical analysis.  Any pointer to debug
> such a thing further?

I think these are well within the bounds that other ports accept.  If
there are genuine bugs, the results are off by thousands or millions of
ulps.

Thanks,
Florian

Re: state of the ports

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Am Freitag, 2. Februar 2024, 12:28:45 CET schrieb Florian Weimer:
> * Xi Ruoyao:
> 
> > On Fri, 2024-02-02 at 15:51 +0800, Xi Ruoyao wrote:
> >> On Fri, 2024-02-02 at 03:35 +0100, Andreas K. Huettel wrote:
> >> > mips (n64)
> >> >  - mostly math failures
> >> 
> >> They are some sort of 2 ulp vs expected 1 ulp things.  I'm not yet sure
> >> if they are hardware issues or Glibc bugs.
> >
> > Something like:
> >
> >  is      : 3 ulp
> >  accepted: 2 ulp
[..]
> >
> > I'm really not an expert in numerical analysis.  Any pointer to debug
> > such a thing further?

That looks like it can be "fixed" by regenerate-ulps, not like a real bug.

So the real number of fails is likely smaller indeed. \o/

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: state of the ports

[Stafford Horne]

On Fri, Feb 02, 2024 at 03:35:04AM +0100, Andreas K. Huettel wrote:
> So... after all the testing gone into the release I thought I'd summarize a bit
> for the list here - by sorting the glibc ports into "excellent", "good", "medium",
> "bad", and "nobody even tried".
> 
> Feel free to correct my assessment... Most of the underlying info is on the
> 2.39 release wiki page.
> 
> Cheers,
> Andreas
> 
> 
> 1. EXCELLENT 
> ============
> 
> aarch64
>   - tests pass nearly everywhere
>   - on raspi, likely timeouts
> 
> arm-linux-gnueabi (soft-float) 
> arm-linux-gnueabihf (hard-float) 
>   - one mystery test failure (elf/tst-align3)
> 
> loongarch64 
>   - tests pass
> 
> powerpc64
> powerpc64le
>   - tests pass
> 
> s390x
>   - tests pass
> 
> x86-64
>   - tests pass
> 
> x86-64 (x32)
>   - one expected failure, tests pass otherwise
> 
> 
> 2. GOOD
> ============
> 
> powerpc64
> ppc 32bit hf
>   - one failure that indicates something broken (nptl/tst-thread-exit-clobber)
> 
> riscv64
>   - ~5 failures, but there may be timeouts among them
> 
> s390 (31bit)
>   - two failures due to gcc breakage (gmon/tst-gmon-pie*)
> 
> sparc
>   - ~5 failures (!)
> 
> x86 (32bit)
>   - in general tests pass
>   - there may be breakage on old hw (pentium2)
> 
> 
> 3. MEDIUM
> ============
> 
> ARC
>  - 11 failures
> 
> hppa2.0
>  - 12 failures (on ~20 year old hw! that's actually good)
> 
> 
> 4. BAD
> ============
> 
> Debian
>  - OK not really a port :) but it seems there are some machines where
>    broken Debian installs lead to massive amounts of FAIL
> 
> mips (o32)
>  - actually broken right now, it seems, the tests came too late
> 
> mips (n64)
>  - mostly math failures
> 
> riscv32
>  - 37 failures across the board
> 
> 
> 99. NOBODY TRIED
> =================
> 
> alpha
> 
> c-sky
> 
> hppa1.1
> 
> m68k
> 
> microblaze
> 
> mips64 (N32)
> 
> niosii
> 
> openrisc

Hello,

I it unfortunate that openrisc ended up in NOBODY TRIED, I usually try to test
the port for each release.  However, this time the testing mail slipped through
my mail filters.

Note, during "glibc 2.36 - Machine testing." Carlos CCed each maintainer on the
announcement mail.  This makes it easier for us maintainers to know we need to
take action.

Can maintainers be CCed on these announcements going forward?

For 2.39, I see the main only went to the list.

-Stafford

Re: state of the ports

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

> 
> Hello,
> 
> I it unfortunate that openrisc ended up in NOBODY TRIED, I usually try to test
> the port for each release.  However, this time the testing mail slipped through
> my mail filters.
> 
> Note, during "glibc 2.36 - Machine testing." Carlos CCed each maintainer on the
> announcement mail.  This makes it easier for us maintainers to know we need to
> take action.
> 
> Can maintainers be CCed on these announcements going forward?
> 
> For 2.39, I see the main only went to the list.

Sorry about that, I promise to do better next time.

Best
-Andreas

> 
> -Stafford
> 
> 


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: state of the ports

[Xi Ruoyao]

On Fri, 2024-02-02 at 22:40 +0100, Andreas K. Huettel wrote:
> Am Freitag, 2. Februar 2024, 12:28:45 CET schrieb Florian Weimer:
> > * Xi Ruoyao:
> > 
> > > On Fri, 2024-02-02 at 15:51 +0800, Xi Ruoyao wrote:
> > > > On Fri, 2024-02-02 at 03:35 +0100, Andreas K. Huettel wrote:
> > > > > mips (n64)
> > > > >  - mostly math failures
> > > > 
> > > > They are some sort of 2 ulp vs expected 1 ulp things.  I'm not yet sure
> > > > if they are hardware issues or Glibc bugs.
> > > 
> > > Something like:
> > > 
> > >  is      : 3 ulp
> > >  accepted: 2 ulp
> [..]
> > > 
> > > I'm really not an expert in numerical analysis.  Any pointer to debug
> > > such a thing further?
> 
> That looks like it can be "fixed" by regenerate-ulps, not like a real bug.
> 
> So the real number of fails is likely smaller indeed. \o/

An update on this: most of the failures were caused by my hardware:the
mips64r2 specification says FP madd should be unfused but the Loongson
hardware implements it as fused.  Passing the correct -march option to
GCC can fix them.  I'll note this in the wiki...

OTOH some ulps indeed needs an update and the updated values seem same
as the other ports using the generic math implementation (RISC-V and
LoongArch) : 

--- ../sysdeps/mips/mips64/libm-test-ulps	2024-02-23 00:56:46.428819912 +0800
+++ /home/xry111/git-repos/glibc-build/math/NewUlps	2024-02-23 00:59:29.085565020 +0800
@@ -1066,17 +1066,17 @@
 ldouble: 1
 
 Function: "j0":
-double: 2
+double: 3
 float: 9
 ldouble: 2
 
 Function: "j0_downward":
-double: 5
+double: 6
 float: 9
 ldouble: 9
 
 Function: "j0_towardzero":
-double: 6
+double: 7
 float: 9
 ldouble: 9
 
@@ -1146,6 +1146,7 @@
 ldouble: 8
 
 Function: "log":
+double: 1
 float: 1
 ldouble: 1

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: state of the ports

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

> 
> OTOH some ulps indeed needs an update and the updated values seem same
> as the other ports using the generic math implementation (RISC-V and
> LoongArch) : 

Thank you! Pushed to master and 2.39 branch.

> 
> --- ../sysdeps/mips/mips64/libm-test-ulps	2024-02-23 00:56:46.428819912 +0800
> +++ /home/xry111/git-repos/glibc-build/math/NewUlps	2024-02-23 00:59:29.085565020 +0800
> @@ -1066,17 +1066,17 @@
>  ldouble: 1
>  
>  Function: "j0":
> -double: 2
> +double: 3
>  float: 9
>  ldouble: 2
>  
>  Function: "j0_downward":
> -double: 5
> +double: 6
>  float: 9
>  ldouble: 9
>  
>  Function: "j0_towardzero":
> -double: 6
> +double: 7
>  float: 9
>  ldouble: 9
>  
> @@ -1146,6 +1146,7 @@
>  ldouble: 8
>  
>  Function: "log":
> +double: 1
>  float: 1
>  ldouble: 1
> 
> 


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: state of the ports

[Mark Wielaard]

Hi,

On Fri, Feb 02, 2024 at 03:35:04AM +0100, Andreas K. Huettel wrote:
> So... after all the testing gone into the release I thought I'd
> summarize a bit for the list here - by sorting the glibc ports into
> "excellent", "good", "medium", "bad", and "nobody even tried".

We have several buildbots at builder.sourceware.org for glibc which
probably should start sending build failure emails when things break.
https://builder.sourceware.org/buildbot/#/builders?tags=glibc

> 1. EXCELLENT 
> ============
> 
> aarch64
>   - tests pass nearly everywhere
>   - on raspi, likely timeouts

https://builder.sourceware.org/buildbot/#/builders/glibc-debian-arm64
Used to have some tests time out, but since we are running with
TIMEOUTFACTOR=4 is all green:

   4807 PASS
     22 UNSUPPORTED
     16 XFAIL
      2 XPASS

> powerpc64
> powerpc64le
>   - tests pass

https://builder.sourceware.org/buildbot/#/builders/glibc-debian-ppc64
https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-ppc64le

We only do a make subdirs=elf check but those all pass:

glibc-debian-ppc64:
    341 PASS
      5 UNSUPPORTED
      2 XPASS

glibc-fedora-ppc64le:
    338 PASS
      2 UNSUPPORTED
      2 XPASS

> s390x
>   - tests pass

https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-s390x
Likewise, just subdirs=elf:

    336 PASS
      2 UNSUPPORTED
      2 XPASS

For ppc64, ppc64le and s390x we could run a different subset if that
is more useful. The issue really is the nptl subdir, which just takes
a very long time (because it doesn't run any test in parallel?)

> x86-64
>   - tests pass

Fully green on glibc-fedora-x86_64
https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-x86_64

   4945 PASS
    274 UNSUPPORTED
     16 XFAIL
      4 XPASS

But on glibc-debian-testing-x86_64 and glibc-rawhide-x86_64 it
occassionally sees one FAIL: elf/tst-audit10 which times out even with
TIMEOUTFACTOR=4

> riscv64
>   - ~5 failures, but there may be timeouts among them

https://builder.sourceware.org/buildbot/#/builders/glibc-ubuntu-riscv

With TIMEOUTFACTOR=4 there are no (timeout) failures anymore.

   4699 PASS
     24 UNSUPPORTED
     16 XFAIL
      2 XPASS

This is the slowest of them all (takes 2 hours 15 minutes to 3 hours,
depending on how good the ccache is). But we have multiple riscv
boards now thanks to StarFive and one of them is dedicated to glibc.

> sparc
>   - ~5 failures (!)

Seeing even more failures:

      9 FAIL
   4725 PASS
     23 UNSUPPORTED
     17 XFAIL
      2 XPASS

FAIL: elf/tst-audit18
FAIL: elf/tst-pldd
FAIL: elf/tst-ptrguard1
FAIL: math/test-float64x-float128-mul
FAIL: nptl/tst-mutex9
FAIL: nptl/tst-oddstacklimit
FAIL: posix/tst-spawn7-pidfd
FAIL: socket/tst-socket-timestamp
FAIL: stdlib/tst-setcontext2

> x86 (32bit)
>   - in general tests pass
>   - there may be breakage on old hw (pentium2)

Really a i386 VM on x86_64:
https://builder.sourceware.org/buildbot/#/builders/glibc-debian-i386

   4851 PASS
     40 UNSUPPORTED
     16 XFAIL
      6 XPASS

Cheers,

Mark

Re: state of the ports

[Adhemerval Zanella Netto]

On 25/02/24 19:47, Mark Wielaard wrote:
> Hi,
> 
> On Fri, Feb 02, 2024 at 03:35:04AM +0100, Andreas K. Huettel wrote:
>> So... after all the testing gone into the release I thought I'd
>> summarize a bit for the list here - by sorting the glibc ports into
>> "excellent", "good", "medium", "bad", and "nobody even tried".
> 
> We have several buildbots at builder.sourceware.org for glibc which
> probably should start sending build failure emails when things break.
> https://builder.sourceware.org/buildbot/#/builders?tags=glibc

It would be really helpful to proper document this instead of just throwing
a comment on https://sourceware.org/glibc/wiki/Buildbot saying the current
information is outdated. The wiki the main source of developer information
and I don't think linking to a blog post if the best way to document it.

And although the buildbot is a good improvement for glibc development,
this does seems like disjoint initiative that did not get much feedback
from glibc community and it is not really tied to current practice. 

On the weekly call [1], we focus most of patchwork [2] results, which has 
results for pre-commit buildbot setup by RedHat and extended by Linaro 
for ARM.  And on each release and on the following weeks, we check and 
discuss the release wiki information to check for potential regressions 
and such. Joseph also keeps a buildbot using build-many-glibcs.py to keep 
track of build for all supported architectures [4].

So I personally think it would be more valuable to hook the buildbots on 
the pre-commit CI instead of doing pos-commit checks.

[1] https://sourceware.org/glibc/wiki/PatchworkReviewMeetings
[2] https://patchwork.sourceware.org/project/glibc/list/
[3] https://sourceware.org/glibc/wiki/Release/2.39
[4] https://sourceware.org/pipermail/libc-testresults/

> 
>> 1. EXCELLENT 
>> ============
>>
>> aarch64
>>   - tests pass nearly everywhere
>>   - on raspi, likely timeouts
> 
> https://builder.sourceware.org/buildbot/#/builders/glibc-debian-arm64
> Used to have some tests time out, but since we are running with
> TIMEOUTFACTOR=4 is all green:
> 
>    4807 PASS
>      22 UNSUPPORTED
>      16 XFAIL
>       2 XPASS
> 
>> powerpc64
>> powerpc64le
>>   - tests pass
> 
> https://builder.sourceware.org/buildbot/#/builders/glibc-debian-ppc64
> https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-ppc64le
> 
> We only do a make subdirs=elf check but those all pass:
> 
> glibc-debian-ppc64:
>     341 PASS
>       5 UNSUPPORTED
>       2 XPASS
> 
> glibc-fedora-ppc64le:
>     338 PASS
>       2 UNSUPPORTED
>       2 XPASS
> 
>> s390x
>>   - tests pass
> 
> https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-s390x
> Likewise, just subdirs=elf:
> 
>     336 PASS
>       2 UNSUPPORTED
>       2 XPASS
> 
> For ppc64, ppc64le and s390x we could run a different subset if that
> is more useful. The issue really is the nptl subdir, which just takes
> a very long time (because it doesn't run any test in parallel?)
> 
>> x86-64
>>   - tests pass
> 
> Fully green on glibc-fedora-x86_64
> https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-x86_64
> 
>    4945 PASS
>     274 UNSUPPORTED
>      16 XFAIL
>       4 XPASS
> 
> But on glibc-debian-testing-x86_64 and glibc-rawhide-x86_64 it
> occassionally sees one FAIL: elf/tst-audit10 which times out even with
> TIMEOUTFACTOR=4
> 
>> riscv64
>>   - ~5 failures, but there may be timeouts among them
> 
> https://builder.sourceware.org/buildbot/#/builders/glibc-ubuntu-riscv
> 
> With TIMEOUTFACTOR=4 there are no (timeout) failures anymore.
> 
>    4699 PASS
>      24 UNSUPPORTED
>      16 XFAIL
>       2 XPASS
> 
> This is the slowest of them all (takes 2 hours 15 minutes to 3 hours,
> depending on how good the ccache is). But we have multiple riscv
> boards now thanks to StarFive and one of them is dedicated to glibc.
> 
>> sparc
>>   - ~5 failures (!)
> 
> Seeing even more failures:
> 
>       9 FAIL
>    4725 PASS
>      23 UNSUPPORTED
>      17 XFAIL
>       2 XPASS
> 
> FAIL: elf/tst-audit18
> FAIL: elf/tst-pldd
> FAIL: elf/tst-ptrguard1
> FAIL: math/test-float64x-float128-mul
> FAIL: nptl/tst-mutex9
> FAIL: nptl/tst-oddstacklimit
> FAIL: posix/tst-spawn7-pidfd
> FAIL: socket/tst-socket-timestamp
> FAIL: stdlib/tst-setcontext2
> 
>> x86 (32bit)
>>   - in general tests pass
>>   - there may be breakage on old hw (pentium2)
> 
> Really a i386 VM on x86_64:
> https://builder.sourceware.org/buildbot/#/builders/glibc-debian-i386
> 
>    4851 PASS
>      40 UNSUPPORTED
>      16 XFAIL
>       6 XPASS
> 
> Cheers,
> 
> Mark