Buildbot failure in Wildebeest Builder on whole buildset

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/898

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/688

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/900

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Mon, Feb 07, 2022 at 08:27:47PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/900
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Frank Ch. Eigler <fche@redhat.com>
> 
> BUILD FAILED: failed test (failure)

This is an odd one. It happens during make distcheck, after make check passed.

FAIL: run-readelf-compressed-zstd.sh
====================================
/srv/buildbot/worker/elfutils-fedora-x86_64/build/elfutils-0.186/_build/sub/src/readelf: failed reading 'hello_i386.ko.zst': zstd decompression failed

Which would mean that zstd decompresssion only failed when using
-fsanitize=undefined and/or running under valgrind (both are enabled
by distcheck, but aren't used during the previous make check).

If someone can replicate this that would be useful.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi all,

On Mon, Feb 07, 2022 at 09:46:26PM +0100, Mark Wielaard wrote:
> This is an odd one. It happens during make distcheck, after make check passed.
> 
> FAIL: run-readelf-compressed-zstd.sh
> ====================================
> /srv/buildbot/worker/elfutils-fedora-x86_64/build/elfutils-0.186/_build/sub/src/readelf: failed reading 'hello_i386.ko.zst': zstd decompression failed
> 
> Which would mean that zstd decompresssion only failed when using
> -fsanitize=undefined and/or running under valgrind (both are enabled
> by distcheck, but aren't used during the previous make check).

Replicated on an updated Fedora 35, run-readelf-compressed-zstd.sh
passes without valgrind, but fails running under valgrind.

Specifically:
$ cp tests/hello_i386.ko.bz2 .
$ bunzip2 hello_i386.ko.bz2
$ zstd hello_i386.ko
$ LD_LIBRARY_PATH=libelf:libdw src/readelf -a hello_i386.ko.zst > /dev/null
$ LD_LIBRARY_PATH=libelf:libdw valgrind -q src/readelf -a hello_i386.ko.zst
src/readelf: failed reading 'hello_i386.ko.zst': zstd decompression failed

I haven't figured out why yet.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Mon, Feb 07, 2022 at 11:29:34PM +0100, Mark Wielaard wrote:
> On Mon, Feb 07, 2022 at 09:46:26PM +0100, Mark Wielaard wrote:
> > This is an odd one. It happens during make distcheck, after make check passed.
> > 
> > FAIL: run-readelf-compressed-zstd.sh
> > ====================================
> > /srv/buildbot/worker/elfutils-fedora-x86_64/build/elfutils-0.186/_build/sub/src/readelf: failed reading 'hello_i386.ko.zst': zstd decompression failed
> > 
> > Which would mean that zstd decompresssion only failed when using
> > -fsanitize=undefined and/or running under valgrind (both are enabled
> > by distcheck, but aren't used during the previous make check).
> 
> Replicated on an updated Fedora 35, run-readelf-compressed-zstd.sh
> passes without valgrind, but fails running under valgrind.
> 
> Specifically:
> $ cp tests/hello_i386.ko.bz2 .
> $ bunzip2 hello_i386.ko.bz2
> $ zstd hello_i386.ko
> $ LD_LIBRARY_PATH=libelf:libdw src/readelf -a hello_i386.ko.zst > /dev/null
> $ LD_LIBRARY_PATH=libelf:libdw valgrind -q src/readelf -a hello_i386.ko.zst
> src/readelf: failed reading 'hello_i386.ko.zst': zstd decompression failed
> 
> I haven't figured out why yet.

Still not much progress. Under valgrind ZSTD_decompressStream does indeed
return an error:

ZSTD_decompressStream: -20 (1)
Corrupted block detected

Will have to trace that with/without valgrind to figure out if it is a
bug in valgrind or some memory operation that behaves different under
valgrind that causes an issue in ZSTD_decompressStream.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/889

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/884

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/880

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/884

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Alexander Kanavin <alex@linutronix.de>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/876

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Alexander Kanavin <alex@linutronix.de>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Thu, 2021-12-16 at 01:10 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/884
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Alexander Kanavin <alex@linutronix.de>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a new failure on builder
> elfutils-fedora-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/876
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Alexander Kanavin <alex@linutronix.de>
> 
> BUILD FAILED: failed test (failure)

So this is really unfortunate and has nothing to do with the patch from
Alexander.

These are two different, but related failures.

On centos-x86_64 this is:

FAIL: run-backtrace-native-core-biarch.sh
=========================================

/usr/bin/coredumpctl
0xf77ac000	0xf77ad000	linux-gate.so.1
0xf77ad000	0xf77d08fc	ld-linux.so.2
0xf75b4000	0xf777ea1c	libc.so.6
0xf777f000	0xf7799248	libpthread.so.0
0x5658e000	0x56591050	backtrace-child-biarch
TID 24658:
# 0 0xf77ac430    	__kernel_vsyscall
# 1 0xf778dd16 - 1	raise
# 2 0x5658eafc - 1	sigusr2
# 3 0x5658ebeb - 1	stdarg
# 4 0x5658ec2f - 1	backtracegen
# 5 0x5658ec38 - 1	start
# 6 0xf7785bbc - 1	start_thread
# 7 0xf76b227e - 1	__clone
TID 24656:
# 0 0xf76b2268    	__clone
/srv/buildbot/worker/elfutils-centos-x86_64/build/tests/backtrace:
dwfl_thread_getframes: No DWARF information found
backtrace: backtrace.c:81: callback_verify: Assertion `seen_main'
failed.
./test-subr.sh: line 84: 24682 Aborted                 (core dumped)
LD_LIBRARY_PATH="${built_library_path}${LD_LIBRARY_PATH:+:}$LD_LIBRARY_
PATH" $VALGRIND_CMD "$@"
backtrace-child-biarch-core.24656: no main

Note that this is a i386 process being backtraced on x86_64.

On fedora-x86_64 this is:


FAIL: run-backtrace-native-core.sh
==================================

/usr/bin/coredumpctl
0x7ffd3d934000	0x7ffd3d935000	linux-vdso.so.1
0x7f4ccbf99000	0x7f4ccbfcd200	ld-linux-x86-64.so.2
0x7f4ccbd7b000	0x7f4ccbf84ad0	libc.so.6
0x56038dbfc000	0x56038dc000a8	backtrace-child
TID 3043057:
# 0 0x7f4ccbe0a89c    	__pthread_kill_implementation
# 1 0x7f4ccbdbd6b6 - 1	raise
# 2 0x56038dbfd3fd - 1	sigusr2
# 3 0x56038dbfd4ca - 1	stdarg
# 4 0x56038dbfd4e0 - 1	backtracegen
# 5 0x56038dbfd4e9 - 1	start
# 6 0x7f4ccbe08ad7 - 1	start_thread
# 7 0x7f4ccbe8d770 - 1	__clone3
TID 3043052:
# 0 0x7f4ccbe8d75d    	__clone3
/srv/buildbot/worker/elfutils-fedora-x86_64/build/tests/backtrace:
dwfl_thread_getframes: address out of range
backtrace: backtrace.c:81: callback_verify: Assertion `seen_main'
failed.
./test-subr.sh: line 84: 3043062 Aborted                 (core dumped)
LD_LIBRARY_PATH="${built_library_path}${LD_LIBRARY_PATH:+:}$LD_LIBRARY_
PATH" $VALGRIND_CMD "$@"
backtrace-child-core.3043052: no main
rmdir: failed to remove 'test-3043029': Directory not empty
FAIL run-backtrace-native-core.sh (exit status: 1)

This is an x86_64 process core being backtraced on x86_64.

The problem in both cases is that the parent cannot unwind from the
exact pc it is stuck at. With eu-stack -v --core we can see (for the
parent TID):

TID 3043052:
#0  0x00007f4ccbe8d75d     __clone3 - libc.so.6
    ../sysdeps/unix/sysv/linux/x86_64/clone3.S:62
eu-stack: dwfl_thread_getframes tid 3043052 at 0x7f4ccbe8d75d in
libc.so.6: address out of range

That is this source code:

ENTRY (__clone3)
        /* Sanity check arguments.  */
        movl    $-EINVAL, %eax
        test    %RDI_LP, %RDI_LP        /* No NULL cl_args pointer.  */
        jz      SYSCALL_ERROR_LABEL
        test    %RDX_LP, %RDX_LP        /* No NULL function pointer.  */
        jz      SYSCALL_ERROR_LABEL

        /* Save the cl_args pointer in R8 which is preserved by the
           syscall.  */
        mov     %RCX_LP, %R8_LP

        /* Do the system call.  */
        movl    $SYS_ify(clone3), %eax

        /* End FDE now, because in the child the unwind info will be
           wrong.  */
        cfi_endproc
        syscall

=>      test    %RAX_LP, %RAX_LP
        jl      SYSCALL_ERROR_LABEL
        jz      L(thread_start)

        ret

L(thread_start):
        cfi_startproc
        /* Clearing frame pointer is insufficient, use CFI.  */
        cfi_undefined (rip)
        /* Clear the frame pointer.  The ABI suggests this be done, to mark
           the outermost frame obviously.  */
        xorl    %ebp, %ebp

        /* Align stack to 16 bytes per the x86-64 psABI.  */
        and     $-16, %RSP_LP
[...]

So the PC is right after the syscall, when as the code says there is no
CFI. Apparently the child ran first and quickly got to the terminating
kill, while the parent was still stuck in the syscall (or just out of
it, but not yet returned from the clone3 call.

I think some synchronization is missed between the parent and child.
But the test code is fairly complex.

Cheers,

Mark 

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/881

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Fri, Dec 10, 2021 at 07:56:05PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/881
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

This is the second time this week the centos-x86_64 builder hung (or
at least didn't produce any output for some time) during make
check. Frank rebuild it and then it all looked fine.

The missing testcase in the both cases was
run-debuginfod-federation-metrics.sh

But I have been unable to make that testcase hang when running make
check by hand.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/868

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, Dec 08, 2021 at 05:20:29PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/868
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

This was a make check timeout. I have been unable to replicate it. A
rebuild on the same buildbot worker passed.

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/860

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Mon, 2021-11-15 at 10:56 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/860
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

I am not sure why this failed. The make check timed out because there
wasn't any output for 20 minutes. Even though this is with valgrind no
test really should take that long.

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/851

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/844

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Thu, 2021-11-04 at 20:14 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/851
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

This was sadly a spurious test. In some cases on this older centos7
setup the biarch (32bit) multi-threaded backtrace from a core file is
missing the "main" frame. But it is somehow timeing related. Maybe the
core file itself is bad? It didn't repeat again and the builder is
green again.

> Build Reason: <unknown>
> Blamelist: Frank Ch. Eigler <fche@redhat.com>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a new failure on builder
> elfutils-fedora-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/844
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-x86_64

This was funny. The fedora-x86_64 builder does a make distcheck. The
distcheck noticed the top of NEWS (first 15 lines) didn't contain the
version number anymore and assumed we forgot to update NEWS for the
release. But it actually was all the new NEWS after the last release
that made the last version number go "out of scope". Frank fixed it by
adding the version number at the top of the NEWS file again.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/841

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/841

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/839

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/807

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64le while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/792

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Mon, 2021-10-18 at 13:45 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-debian-
> amd64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/841
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64
> 
> Build Reason: <unknown>
> Blamelist: Mark Wielaard <mark@klomp.org>
> 
> BUILD FAILED: failed test (failure)

I already saw this earlier and it is resolved by:

commit e3e70782a1d1a246844215991bbd2d6b60d0aa41
Author: Mark Wielaard <mark@klomp.org>
Date:   Mon Oct 18 14:35:20 2021 +0200

    libdw: Don't pass NULL to dwarf_peel_type

https://builder.wildebeest.org/buildbot/#/changes/6894

All green again.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/839

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/798

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/623

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/264

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Thu, 2021-09-16 at 15:00 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/839
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)

So it looks like half of the buildbot workers got a failure on the new
run-debuginfod-response-headers.sh testcase grepping for Content-Length 
in the vlog-find$PORT1.1 output file from the debuginfod-find test.

See https://builder.wildebeest.org/buildbot/#/changes/2659

I don't immediately know what they have in common.
My first thought was that it might depend on the libmicrohttpd version?
But it seems the failing workers are both new and old distros (centos,
debian and fedora).

Noah, could you take a look to see what/why the testcase might be
failing?

Thanks,

Mark

> Sincerely,
>  -The BuildbotThe Buildbot has detected a new failure on builder
> elfutils-fedora-s390x while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/798
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a new failure on builder
> elfutils-debian-armhf while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/15/builds/623
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-armhf
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a new failure on builder
> elfutils-debian-arm64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/45/builds/264
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-arm64
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The Buildbot
> 

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/795

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sun, Sep 12, 2021 at 11:16:09PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/795
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x

This is the same failure we saw on fedora-ppc64 and centos-x86_64 yesterday.
https://builder.wildebeest.org/buildbot/#/builders/10/builds/795/steps/8/logs/test-suite_log

I still don't understand why. In the logs we can see (for the PORT2 server):

[Sun Sep 12 22:56:26 2021] (1493056/1493066): recorded buildid=a0a48245eb29786f7b6853df68ab23cb608b344b file=/home/mjw/bb/wildebeest/elfutils-fedora-s390x/build/tests/dwfllines mtime=1631486319 atype=ED

But then, 2 seconds later:
[Sun Sep 12 22:56:28 2021] (1493056/1493388): searching for buildid=a0a48245eb29786f7b6853df68ab23cb608b344b artifacttype=debuginfo suffix=
[Sun Sep 12 22:56:28 2021] (1493056/1493388): not found
[Sun Sep 12 22:56:28 2021] (1493056/1493388): 127.0.0.1:47886 UA:elfutils/0.185,Linux/s390x,fedora/34 XFF: GET /buildid/a0a48245eb29786f7b6853df68ab23cb608b344b/debuginfo 404 9 0+2ms

Somewhere inbetween the buildid seems to have been forgotten. But I
cannot figure out why or where. It is clearly non-deterministic since
normally the tests PASS.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2693 bytes --]

Hi,

On Mon, 2021-09-13 at 11:06 +0200, Mark Wielaard wrote:
> On Sun, Sep 12, 2021 at 11:16:09PM +0000, 
> buildbot@builder.wildebeest.org wrote:
> > The Buildbot has detected a new failure on builder elfutils-fedora-
> > s390x while building elfutils.
> > Full details are available at:
> >     https://builder.wildebeest.org/buildbot/#builders/10/builds/795
> > 
> > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > 
> > Worker for this Build: fedora-s390x
> 
> This is the same failure we saw on fedora-ppc64 and centos-x86_64
> yesterday.
> https://builder.wildebeest.org/buildbot/#/builders/10/builds/795/steps/8/logs/test-suite_log
> 
> I still don't understand why. In the logs we can see (for the PORT2
> server):
> 
> [Sun Sep 12 22:56:26 2021] (1493056/1493066): recorded
> buildid=a0a48245eb29786f7b6853df68ab23cb608b344b
> file=/home/mjw/bb/wildebeest/elfutils-fedora-
> s390x/build/tests/dwfllines mtime=1631486319 atype=ED
> 
> But then, 2 seconds later:
> [Sun Sep 12 22:56:28 2021] (1493056/1493388): searching for
> buildid=a0a48245eb29786f7b6853df68ab23cb608b344b
> artifacttype=debuginfo suffix=
> [Sun Sep 12 22:56:28 2021] (1493056/1493388): not found
> [Sun Sep 12 22:56:28 2021] (1493056/1493388): 127.0.0.1:47886
> UA:elfutils/0.185,Linux/s390x,fedora/34 XFF: GET
> /buildid/a0a48245eb29786f7b6853df68ab23cb608b344b/debuginfo 404 9
> 0+2ms
> 
> Somewhere inbetween the buildid seems to have been forgotten. But I
> cannot figure out why or where. It is clearly non-deterministic since
> normally the tests PASS.

So the issue is triggered by this part in groom ():

   // delete buildids with no references in _r_de or _f_de tables;
   // cascades to _r_sref & _f_s records
   sqlite_ps buildids_del (db, "nuke orphan buildids",
                           "delete from " BUILDIDS "_buildids "
                           "where not exists (select 1 from " BUILDIDS "_f_de d where " BUILDIDS "_buildids.id = d.buildid) "
                           "and not exists (select 1 from " BUILDIDS "_r_de d where " BUILDIDS "_buildids.id = d.buildid)");
   buildids_del.reset().step_ok_done();

When commenting that out I can run the tests (or a simplified version
using just one server and on one client request as attached) 30000
times without issue. While with groom executing that part of the 
code the test will fail after a couple hundred cycles.

Now the question is whether it is reasonable that groom removes the
buildid here. Is that because of the way the test is written? Or is
this a real bug where there is a bad interaction between a (partial?)
scan run and a groom cycle?

Cheers,

Mark


[-- Attachment #2: run-debuginfod-federation-link.sh --]
[-- Type: application/x-shellscript, Size: 2315 bytes --]

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/835

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Colin Cross <ccross@google.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/777

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Colin Cross <ccross@google.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Sun, Sep 12, 2021 at 09:56:39PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/835
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Colin Cross <ccross@google.com>
> 
> BUILD FAILED: failed test (failure)
> [...]
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/12/builds/777

This is certainly not caused by Colin's patch.  It is the same failure
on two different systems, in related, but different tests.

run-debuginfod-federation-link.sh
run-debuginfod-federation-metrics.sh
run-debuginfod-federation-sqlite.sh

All have the following subtest:

# confirm that first server can't resolve symlinked info in L/ but second can
BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
         -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
file L/foo
file -L L/foo
export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
rm -rf $DEBUGINFOD_CACHE_PATH
testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID

Where L/foo is created with:

ln -s ${abs_builddir}/dwfllines L/foo   # any program not used elsewhere in this test

Both debuginfod servers have L in their scan path, but only the one on
PORT2 has -L L. So the one on PORT1 should not resolve the foo
symlink, but the one on PORT2 doesn't.

For some reason it seems even the server on PORT2 doesn't record L/foo
sometimes (but not always).

I haven't figured out why. Both failures above have extended logs, so
maybe someone else can figure out what is going wrong in the failure
case?

https://builder.wildebeest.org/buildbot/#/builders/12/builds/777/steps/8/logs/test-suite_log
https://builder.wildebeest.org/buildbot/#/builders/1/builds/835/steps/8/logs/test-suite_log

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/829

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/819

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/823

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/789

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64le while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/774

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/772

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/614

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/255

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/818

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/823

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/815

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/813

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/607

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/814

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/810

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/763

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/603

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Fri, Sep 03, 2021 at 02:08:40PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/15/builds/603
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-armhf
> 
> Build Reason: <unknown>
> Blamelist: Mark Wielaard <mark@klomp.org>
> 
> BUILD FAILED: failed test (failure)

So some of the builders did pass without any failures. But some had
errors, and not all the same ones. I couldn't replicate locally
(everything was green) both with make check and make distcheck.

But looking at the tests I noticed not all used a clean new cache and
some used the default database.

So hopefully the attached patch makes things a bit more stable by
explicitly setting the DEBUGINFOD_CACHE_PATH and invoke debuginfod
with a fresh sqlite database.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Make-sure-all-debuginfod-tests-use-a-clean-dat.patch --]
[-- Type: text/x-diff, Size: 10191 bytes --]

From 8e9edeab19931f878e16cc1bff6ac286937e997f Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 4 Sep 2021 01:40:53 +0200
Subject: [PATCH] tests: Make sure all debuginfod tests use a clean database
 and cache.

Always set DEBUGINFOD_CACHE_PATH to an unique (new) directory
and make sure that each debuginfod invocation uses a new sqlite
database.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog                            | 15 +++++++++++++++
 tests/run-debuginfod-000-permission.sh     |  3 ++-
 tests/run-debuginfod-archive-groom.sh      |  3 ++-
 tests/run-debuginfod-archive-test.sh       |  5 ++++-
 tests/run-debuginfod-dlopen.sh             |  7 ++++++-
 tests/run-debuginfod-duplicate-urls.sh     |  4 ++++
 tests/run-debuginfod-extraction.sh         |  6 +++++-
 tests/run-debuginfod-fd-prefetch-caches.sh |  6 +++++-
 tests/run-debuginfod-malformed.sh          |  3 ++-
 tests/run-debuginfod-sizetime.sh           |  6 +++++-
 tests/run-debuginfod-tmp-home.sh           |  3 ++-
 tests/run-debuginfod-writable.sh           |  6 +++++-
 12 files changed, 57 insertions(+), 10 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 9540f738..178697bb 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,18 @@
+2021-09-03  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-000-permission.sh: Set DEBUGINFOD_CACHE_PATH
+	and use an unique sqlite db.
+	* run-debuginfod-archive-groom.sh: Likewise.
+	* run-debuginfod-archive-test.sh: Likewise.
+	* run-debuginfod-dlopen.sh: Likewise.
+	* run-debuginfod-duplicate-urls.sh: Likewise.
+	* run-debuginfod-extraction.sh: Likewise.
+	* run-debuginfod-fd-prefetch-caches.sh: Likewise.
+	* run-debuginfod-malformed.sh: Likewise.
+	* run-debuginfod-sizetime.sh: Likewise.
+	* run-debuginfod-tmp-home.sh: Likewise.
+	* run-debuginfod-writable.sh: Likewise.
+
 2021-09-03  Mark Wielaard  <mark@klomp.org>
 
 	* debuginfod-subr.sh (EXTRA_DIST): Add debuginfod-subr.sh.
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
index 28e54385..1e92bdb8 100755
--- a/tests/run-debuginfod-000-permission.sh
+++ b/tests/run-debuginfod-000-permission.sh
@@ -22,12 +22,13 @@
 set -x
 unset VALGRIND_CMD
 DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
 export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 # This variable is essential and ensures no time-race for claiming ports occurs
 # set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
 base=8000
 get_ports
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -d $DB \
     -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
index 0ee056ff..4b991f53 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -26,6 +26,7 @@ unset VALGRIND_CMD
 base=8100
 get_ports
 DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
 export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 export DEBUGINFOD_TIMEOUT=10
 export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
@@ -68,7 +69,7 @@ if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
-tempfiles vlog3 $DB
+tempfiles vlog3
 cp -rvp ${abs_srcdir}/debuginfod-tars Z
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
index bc500540..1ec57bba 100755
--- a/tests/run-debuginfod-archive-test.sh
+++ b/tests/run-debuginfod-archive-test.sh
@@ -28,8 +28,11 @@ mkdir R
 # set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
 base=8300
 get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -d $DB -t0 -g0 -v R > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
 errfiles vlog$PORT1
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
index 6476612c..0f2d13e2 100755
--- a/tests/run-debuginfod-dlopen.sh
+++ b/tests/run-debuginfod-dlopen.sh
@@ -27,7 +27,12 @@ mkdir F
 # set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
 base=8500
 get_ports
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 -d $DB \
     -t0 -g0 -v F > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
index b76b39a3..50e39cb2 100755
--- a/tests/run-debuginfod-duplicate-urls.sh
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -26,6 +26,10 @@ unset VALGRIND_CMD
 base=8600
 get_ports
 
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
 env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
     $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
 PID1=$!
diff --git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
index 7c534d09..e984f32f 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -28,7 +28,11 @@ mkdir R Z
 base=8700
 get_ports
 
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
 errfiles vlog$PORT1
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
index 08b32923..61fee9e9 100755
--- a/tests/run-debuginfod-fd-prefetch-caches.sh
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -31,8 +31,12 @@ PREFETCH_MBS=100
 base=8800
 get_ports
 
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
 echo $PORT1
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \
     --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
     --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
 PID1=$!
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
index 78b3b7fc..eb504181 100755
--- a/tests/run-debuginfod-malformed.sh
+++ b/tests/run-debuginfod-malformed.sh
@@ -28,11 +28,12 @@ base=9200
 # find an unused port number
 get_ports
 DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
 export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 mkdir F
 mkdir -p $DEBUGINFOD_CACHE_PATH
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -d $DB \
     -t0 -g0 -v F > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
index 068b548b..2cf6f252 100755
--- a/tests/run-debuginfod-sizetime.sh
+++ b/tests/run-debuginfod-sizetime.sh
@@ -25,6 +25,10 @@ unset VALGRIND_CMD
 base=9600
 get_ports
 
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
 echo "int main() { return 0; }" > ${PWD}/prog.c
 # Create a subdirectory to confound source path names
 mkdir foobar
@@ -34,7 +38,7 @@ tempfiles prog prog.debug prog.c
 BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
           -a prog | grep 'Build ID' | cut -d ' ' -f 7`
 
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -d $DB -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
 errfiles  vlog$PORT1
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
index 81986198..25b89be4 100755
--- a/tests/run-debuginfod-tmp-home.sh
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -23,6 +23,7 @@ set -x
 unset VALGRIND_CMD
 
 DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
 export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 
 mkdir F
@@ -32,7 +33,7 @@ mkdir -p $DEBUGINFOD_CACHE_PATH
 base=9700
 get_ports
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -d $DB \
     -t0 -g0 -v F > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
index 440a5666..0bc70379 100755
--- a/tests/run-debuginfod-writable.sh
+++ b/tests/run-debuginfod-writable.sh
@@ -28,7 +28,11 @@ mkdir F
 base=9800
 get_ports
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -d $DB \
     -t0 -g0 -v F > vlog$PORT1 2>&1 &
 PID1=$!
 tempfiles vlog$PORT1
-- 
2.32.0

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, 2021-09-04 at 01:47 +0200, Mark Wielaard wrote:
> On Fri, Sep 03, 2021 at 02:08:40PM +0000, 
> buildbot@builder.wildebeest.org wrote:
> > The Buildbot has detected a new failure on builder elfutils-debian-
> > armhf while building elfutils.
> > Full details are available at:
> >     https://builder.wildebeest.org/buildbot/#builders/15/builds/603
> > 
> > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > 
> > Worker for this Build: debian-armhf
> > 
> > Build Reason: <unknown>
> > Blamelist: Mark Wielaard <mark@klomp.org>
> > 
> > BUILD FAILED: failed test (failure)
> 
> So some of the builders did pass without any failures. But some had
> errors, and not all the same ones. I couldn't replicate locally
> (everything was green) both with make check and make distcheck.
> 
> But looking at the tests I noticed not all used a clean new cache and
> some used the default database.
> 
> So hopefully the attached patch makes things a bit more stable by
> explicitly setting the DEBUGINFOD_CACHE_PATH and invoke debuginfod
> with a fresh sqlite database.

Sadly we still have failures even with this patch:
https://builder.wildebeest.org/buildbot/#/changes/2567

The good news is that elfutils-fedora-x86_64, elfutils-fedora-ppc64le,
elfutils-fedora-s390x, elfutils-debian-i386, elfutils-fedora-ppc64 and
elfutils-debian-armhf are fully green, but...

On elfutils-centos-x86_64 make check fails with:

FAIL: run-debuginfod-tmp-home.sh
FAIL: run-debuginfod-writable.sh

On elfutils-debian-arm64 make check fails with:

FAIL: run-debuginfod-federation-sqlite.sh

On elfutils-centos-x86_64 make check first timed out (!?!)
then it failed with:

FAIL: run-debuginfod-tmp-home.sh
FAIL: run-debuginfod-writable.sh

On elfutils-debian-amd64 make check failed with:

FAIL: run-debuginfod-archive-groom.sh

The failures still look somewhat random (although less random than
before). I don't yet know why.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

Hi,

On Mon, Sep 06, 2021 at 05:07:50PM +0200, Mark Wielaard wrote:
> On Sat, 2021-09-04 at 01:47 +0200, Mark Wielaard wrote:
> > On Fri, Sep 03, 2021 at 02:08:40PM +0000, 
> > buildbot@builder.wildebeest.org wrote:
> > > The Buildbot has detected a new failure on builder elfutils-debian-
> > > armhf while building elfutils.
> > > Full details are available at:
> > >     https://builder.wildebeest.org/buildbot/#builders/15/builds/603
> > > 
> > > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > > 
> > > Worker for this Build: debian-armhf
> > > 
> > > Build Reason: <unknown>
> > > Blamelist: Mark Wielaard <mark@klomp.org>
> > > 
> > > BUILD FAILED: failed test (failure)
> > 
> > So some of the builders did pass without any failures. But some had
> > errors, and not all the same ones. I couldn't replicate locally
> > (everything was green) both with make check and make distcheck.
> > 
> > But looking at the tests I noticed not all used a clean new cache and
> > some used the default database.
> > 
> > So hopefully the attached patch makes things a bit more stable by
> > explicitly setting the DEBUGINFOD_CACHE_PATH and invoke debuginfod
> > with a fresh sqlite database.
> 
> Sadly we still have failures even with this patch:
> https://builder.wildebeest.org/buildbot/#/changes/2567
> 
> The good news is that elfutils-fedora-x86_64, elfutils-fedora-ppc64le,
> elfutils-fedora-s390x, elfutils-debian-i386, elfutils-fedora-ppc64 and
> elfutils-debian-armhf are fully green, but...
> 
> On elfutils-centos-x86_64 make check fails with:
> 
> FAIL: run-debuginfod-tmp-home.sh
> FAIL: run-debuginfod-writable.sh
> 
> On elfutils-debian-arm64 make check fails with:
> 
> FAIL: run-debuginfod-federation-sqlite.sh
> 
> On elfutils-centos-x86_64 make check first timed out (!?!)
> then it failed with:
> 
> FAIL: run-debuginfod-tmp-home.sh
> FAIL: run-debuginfod-writable.sh
> 
> On elfutils-debian-amd64 make check failed with:
> 
> FAIL: run-debuginfod-archive-groom.sh
> 
> The failures still look somewhat random (although less random than
> before). I don't yet know why.

This doesn't really explain the above failures, but there were still
two tests that didn't use their own debuginfod client cache. Also one
set a useless, but confusing DEBUGINFOD_URLS.

Still looking for explanations of the other FAILS (which I cannot
reproduce locally).

Cheers,

Mark

[-- Attachment #2: 0001-tests-Set-DEBUGINFOD_CACHE_PATH-for-run-debuginfod-f.patch --]
[-- Type: text/x-diff, Size: 2926 bytes --]

From 44a36e39fc7df2a3e2f9f278f79f65233d774a48 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 6 Sep 2021 21:48:38 +0200
Subject: [PATCH] tests: Set DEBUGINFOD_CACHE_PATH for
 run-debuginfod-{file,query-retry}.sh

---
 tests/ChangeLog                     | 7 +++++++
 tests/run-debuginfod-file.sh        | 6 +++---
 tests/run-debuginfod-query-retry.sh | 2 +-
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 178697bb..c1760877 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,10 @@
+2021-09-06  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-file.sh: Set DEBUGINFOD_CACHE_PATH. Export
+	correct DEBUGINFOD_URLS.
+	* run-debuginfod-query-retry.sh: Add DEBUGINFOD_CACHE_PATH
+	to env.
+
 2021-09-03  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-000-permission.sh: Set DEBUGINFOD_CACHE_PATH
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
index 22c956dd..341bbc68 100755
--- a/tests/run-debuginfod-file.sh
+++ b/tests/run-debuginfod-file.sh
@@ -22,7 +22,7 @@
 set -x
 unset VALGRIND_CMD
 
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/   # or without trailing /
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 
 # Test fetching a file using file:// . No debuginfod server needs to be run for
 # this test.
@@ -30,11 +30,11 @@ local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/sourc
 mkdir -p ${local_dir}
 echo "int main() { return 0; }" > ${local_dir}/main.c
 # first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
+export DEBUGINFOD_URLS=""
 testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
 
 # Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+export DEBUGINFOD_URLS="file://${PWD}/mocktree/"
 filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
 cmp $filename ${local_dir}/main.c
 
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
index 3c5542d5..c9192510 100755
--- a/tests/run-debuginfod-query-retry.sh
+++ b/tests/run-debuginfod-query-retry.sh
@@ -24,7 +24,7 @@ unset VALGRIND_CMD
 
 ########################################################################
 # set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache \
         ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
         | grep -c 'Retry failed query'`
 if [ $retry_attempts -ne 10 ]; then
-- 
2.32.0

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On Mon, Sep 06, 2021 at 10:02:11PM +0200, Mark Wielaard wrote:
> This doesn't really explain the above failures, but there were still
> two tests that didn't use their own debuginfod client cache. Also one
> set a useless, but confusing DEBUGINFOD_URLS.
> 
> Still looking for explanations of the other FAILS (which I cannot
> reproduce locally).

As expected there are still several failures on the buildbot workers,
but I did manage to replicate one failure locally. It happened because
PORT1 and PORT2 were equal.

    ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break

Does actually break when PORT1 == PORT2. The && -ne should have been
|| -eq. But just simplify the port selection a bit to use
non-overlapping ranges (split the 100 possible ports in 0-49 for port1
and 50-99 for port2).

Also because of a typo on error only the metrics of port1 were
shown. Fix the typo so we can better diagnose the remaining failures.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Print-metrics-for-both-ports-on-error-and-fix-.patch --]
[-- Type: text/x-diff, Size: 2036 bytes --]

From 7880ccb6483e76847cdf1c6c4e45a2180bee820a Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 6 Sep 2021 23:04:06 +0200
Subject: [PATCH] tests: Print metrics for both ports on error and fix port
 selection

On error we would only print the metrics of one port (twice) because
of a typo. Also PORT1 and PORT2 could be equal because of a logic
error. Fix the typo and simplify the port selection by using
non-overlapping ranges to select PORT1 and PORT2.
---
 tests/ChangeLog          | 7 +++++++
 tests/debuginfod-subr.sh | 8 ++++----
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index c1760877..14eb4d98 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,10 @@
+2021-09-06  Mark Wielaard  <mark@klomp.org>
+
+	* debuginfod-subr.sh (err): Change ports to port in for loop so both
+	PORT1 and PORT2 are used.
+	(get_ports): Simplify port selection by using for 50 for PORT1 and
+	second 50 for PORT2.
+
 2021-09-06  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-file.sh: Set DEBUGINFOD_CACHE_PATH. Export
diff --git a/tests/debuginfod-subr.sh b/tests/debuginfod-subr.sh
index 3222a2b0..7d238436 100755
--- a/tests/debuginfod-subr.sh
+++ b/tests/debuginfod-subr.sh
@@ -39,7 +39,7 @@ trap cleanup 0 1 2 3 5 9 15
 errfiles_list=
 err() {
     echo ERROR REPORTS
-    for ports in $PORT1 $PORT2
+    for port in $PORT1 $PORT2
     do
         echo ERROR REPORT $port metrics
         curl -s http://127.0.0.1:$port/metrics
@@ -129,13 +129,13 @@ archive_test() {
 
 get_ports() {
   while true; do
-    PORT1=`expr '(' $RANDOM % 100 ')' + $base`
+    PORT1=`expr '(' $RANDOM % 50 ')' + $base`
     ss -atn | fgrep ":$PORT1" || break
   done
 # Some tests will use two servers, so assign the second var
   while true; do
-    PORT2=`expr '(' $RANDOM % 100 ')' + $base`
-    ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+    PORT2=`expr '(' $RANDOM % 50 ')' + $base + 50`
+    ss -atn | fgrep ":$PORT2" || break
   done
 
 }
-- 
2.32.0

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

Hi,

On Mon, Sep 06, 2021 at 10:02:11PM +0200, Mark Wielaard wrote:
> Still looking for explanations of the other FAILS (which I cannot
> reproduce locally).

I think I finally got it. The thing that all the tests that are
(sometimes) failing have in commen is that they use SIGUSR1 or SIGUSR2
to trigger a new scan or groom cycle. But they don't check whether the
initial scan or groom cycle has finished. So the attached patch
explicitly does that before the first SIGUSR1 or SIGUSR2 is fired at
the debuginfod server.

Pushed,

Mark


[-- Attachment #2: 0001-tests-Make-sure-to-wait-for-initial-scan-and-groom-c.patch --]
[-- Type: text/x-diff, Size: 18080 bytes --]

From dbd4c8766ea80cb0b6294a6755a556017215eb99 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Tue, 7 Sep 2021 01:50:51 +0200
Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle

At startup the debuginfod server does a scan and groom cycle. Make
sure to wait for that before triggering a new one with SIGUSR1 or
SIGURST2.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog                            | 17 +++++++++++++++++
 tests/run-debuginfod-archive-groom.sh      | 13 +++++++++----
 tests/run-debuginfod-archive-rename.sh     | 12 ++++++++----
 tests/run-debuginfod-archive-test.sh       |  4 +++-
 tests/run-debuginfod-artifact-running.sh   |  8 +++++---
 tests/run-debuginfod-dlopen.sh             |  6 +++++-
 tests/run-debuginfod-extraction.sh         |  6 ++++--
 tests/run-debuginfod-federation-link.sh    |  9 +++++++--
 tests/run-debuginfod-federation-metrics.sh |  2 +-
 tests/run-debuginfod-federation-sqlite.sh  | 10 +++++++---
 tests/run-debuginfod-malformed.sh          |  5 ++++-
 tests/run-debuginfod-regex.sh              |  7 +++++--
 tests/run-debuginfod-tmp-home.sh           |  5 ++++-
 tests/run-debuginfod-writable.sh           |  5 ++++-
 14 files changed, 83 insertions(+), 26 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 436a1c45..61080c52 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,20 @@
+2021-09-06  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-archive-groom.sh: Wait for initial scan and groom
+	cycle.
+	* run-debuginfod-archive-rename.sh: Likewise.
+	* run-debuginfod-federation-sqlite.sh: Likewise.
+	* run-debuginfod-archive-test.sh: Wait for initial scan cycle.
+	* run-debuginfod-artifact-running.sh: Likewise.
+	* run-debuginfod-dlopen.sh: Likewise.
+	* run-debuginfod-extraction.sh: Likewise.
+	* run-debuginfod-federation-link.sh: Likewise.
+	* run-debuginfod-federation-metrics.sh: Likewise.
+	* run-debuginfod-malformed.sh: Likewise.
+	* run-debuginfod-regex.sh: Likewise.
+	* run-debuginfod-tmp-home.sh: Likewise.
+	* run-debuginfod-writable.sh: Likewise.
+
 2021-09-06  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-archive-groom.sh: Set DEBUGINFOD_URLS after starting
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
index 1e616448..1e7bd67a 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -57,9 +57,12 @@ cp -rvp ${abs_srcdir}/debuginfod-rpms R
 if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
+
+# wait till the initial scan is done before triggering a new one
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
@@ -72,7 +75,7 @@ tempfiles vlog3
 cp -rvp ${abs_srcdir}/debuginfod-tars Z
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 ########################################################################
@@ -83,7 +86,7 @@ txz=$(find Z -name \*tar.xz | wc -l)
 
 kill -USR1 $PID1  # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
 # Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
@@ -136,9 +139,11 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
 # Drop some of the artifacts, run a groom cycle; confirm that
 # debuginfod has forgotten them, but remembers others
 rm -r R/debuginfod-rpms/rhel6/*
+
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 kill -USR2 $PID1  # groom cycle
 ## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
 # Expect 4 rpms containing 2 buildids to be deleted by the groom
 wait_ready $PORT1 'groomed_total{decision="stale"}' 4
 
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
index 7ad4786e..4fc1b441 100755
--- a/tests/run-debuginfod-archive-rename.sh
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -49,9 +49,11 @@ if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
+# Make sure the initial scan is done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
@@ -63,23 +65,25 @@ SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
 # there are two copies of the same buildid in the index, one for the
 # no-longer-existing file name, and one under the new name.
 
+# Make sure the initial groom cycle has been done
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 # run a groom cycle to force server to drop its fdcache
 kill -USR2 $PID1  # groom cycle
 wait_ready $PORT1 'thread_work_total{role="groom"}' 2
 # move it around a couple of times to make it likely to hit a nonexistent entry during iteration
 mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
 kill -USR1 $PID1  # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
 kill -USR1 $PID1  # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
 kill -USR1 $PID1  # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 5
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
index 1ec57bba..9f7454bc 100755
--- a/tests/run-debuginfod-archive-test.sh
+++ b/tests/run-debuginfod-archive-test.sh
@@ -38,6 +38,8 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 # Server must become ready
 wait_ready $PORT1 'ready' 1
+# And make sure first scan is done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 
 # Be patient when run on a busy machine things might take a bit.
 export DEBUGINFOD_TIMEOUT=10
@@ -55,7 +57,7 @@ fi
 
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
index e96cb966..4eae0200 100755
--- a/tests/run-debuginfod-artifact-running.sh
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -51,9 +51,11 @@ export DEBUGINFOD_TIMEOUT=10
 
 # Server must become ready
 wait_ready $PORT1 'ready' 1
-kill -USR1 $PID1
-
+# And the initial scan should have been done
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
@@ -76,7 +78,7 @@ tempfiles F/prog2 F/prog2.debug
 
 kill -USR1 $PID1
 # Now there should be 3 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
index 0f2d13e2..5f33394a 100755
--- a/tests/run-debuginfod-dlopen.sh
+++ b/tests/run-debuginfod-dlopen.sh
@@ -70,9 +70,13 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 
 mv p+r%o\$g F
 mv p+r%o\$g.debug F
+
+# Make sure the initial scan has finished.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 kill -USR1 $PID1
 # Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
index e984f32f..4750f184 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -54,9 +54,11 @@ fi
 
 cp -rvp ${abs_srcdir}/debuginfod-tars Z
 
+# Make sure the initial scan has finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
@@ -71,7 +73,7 @@ wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
 
 kill -USR1 $PID1  # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
 # Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
index 42b8f101..1aef7174 100755
--- a/tests/run-debuginfod-federation-link.sh
+++ b/tests/run-debuginfod-federation-link.sh
@@ -62,6 +62,9 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 
 mv prog F
 mv prog.debug F
+
+# Make sure initial scan was done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index.
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
@@ -83,9 +86,11 @@ tempfiles ${DB}_2
 
 wait_ready $PORT2 'ready' 1
 
+# Make sure initial scan was done
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID2
 # Wait till both files are in the index.
-wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT2 'thread_busy{role="scan"}' 0
 
@@ -99,7 +104,7 @@ if type bsdtar 2>/dev/null; then
     # copy in the deb files
     cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
     kill -USR1 $PID2
-    wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+    wait_ready $PORT2 'thread_work_total{role="traverse"}' 3
     wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
     wait_ready $PORT2 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
index 2e7550a6..0cc4c2f7 100755
--- a/tests/run-debuginfod-federation-metrics.sh
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -45,7 +45,7 @@ errfiles vlog$PORT1
 
 wait_ready $PORT1 'ready' 1
 
-kill -USR1 $PID1
+# Wait till initial scan is done
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
index a323b98e..5a18b4bb 100755
--- a/tests/run-debuginfod-federation-sqlite.sh
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -59,7 +59,7 @@ errfiles vlog$PORT1
 
 wait_ready $PORT1 'ready' 1
 
-kill -USR1 $PID1
+# Wait till initial scan is done
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
@@ -73,13 +73,17 @@ tempfiles vlog$PORT2
 errfiles vlog$PORT2
 tempfiles ${DB}_2
 wait_ready $PORT2 'ready' 1
+# Wait till initial scan is done
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+# And initial groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 
 export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2 
 if type bsdtar 2>/dev/null; then
     # copy in the deb files
     cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
     kill -USR1 $PID2
-    wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+    wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
     wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
     wait_ready $PORT2 'thread_busy{role="scan"}' 0
 
@@ -177,7 +181,7 @@ wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 kill -USR2 $PID1
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
 curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
 curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
 # Run the tests again without the servers running. The target file should
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
index eb504181..3bc9e799 100755
--- a/tests/run-debuginfod-malformed.sh
+++ b/tests/run-debuginfod-malformed.sh
@@ -40,6 +40,9 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 # Server must become ready
 wait_ready $PORT1 'ready' 1
+# And an initial scan should be done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/   # or without trailing /
 
 # Be patient when run on a busy machine things might take a bit.
@@ -70,7 +73,7 @@ mv p+r%o\$g F
 mv p+r%o\$g.debug F
 kill -USR1 $PID1
 # Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
index 32ddf6ff..6837ea86 100755
--- a/tests/run-debuginfod-regex.sh
+++ b/tests/run-debuginfod-regex.sh
@@ -37,7 +37,10 @@ env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
 PID1=$!
 tempfiles vlog$PORT1
 errfiles vlog$PORT1
+
+# Wait till the server is ready and an initial scan has been done
 wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 
 # Compile a simple program, strip its debuginfo and save the build-id.
 # Also move the debuginfo into another directory so that elfutils
@@ -55,7 +58,7 @@ tempfiles F/prog.debug F/prog
 
 kill -USR1 $PID1
 # Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 cp ${DB} ${DB}.backup
@@ -79,7 +82,7 @@ errfiles vlog$PORT2
 # Server must become ready
 wait_ready $PORT2 'ready' 1
 
-kill -USR2 $PID1
+# Wait till the initial groom cycle has been done
 wait_ready $PORT2 'thread_work_total{role="groom"}' 1
 wait_ready $PORT2 'groom{statistic="archive d/e"}'  0
 wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
index 25b89be4..dc9accb0 100755
--- a/tests/run-debuginfod-tmp-home.sh
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -40,6 +40,9 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 # Server must become ready
 wait_ready $PORT1 'ready' 1
+# And initial scan should be done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/   # or without trailing /
 
 # Be patient when run on a busy machine things might take a bit.
@@ -71,7 +74,7 @@ mv p+r%o\$g F
 mv p+r%o\$g.debug F
 kill -USR1 $PID1
 # Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
index 0bc70379..69ececb3 100755
--- a/tests/run-debuginfod-writable.sh
+++ b/tests/run-debuginfod-writable.sh
@@ -39,6 +39,9 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 # Server must become ready
 wait_ready $PORT1 'ready' 1
+# And initial scan should be done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/   # or without trailing /
 
 # Be patient when run on a busy machine things might take a bit.
@@ -67,7 +70,7 @@ mv p+r%o\$g F
 mv p+r%o\$g.debug F
 kill -USR1 $PID1
 # Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
-- 
2.32.0

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Dmitry V. Levin]

On Tue, Sep 07, 2021 at 02:12:18AM +0200, Mark Wielaard wrote:
[...]
> Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle
> 
> At startup the debuginfod server does a scan and groom cycle. Make
> sure to wait for that before triggering a new one with SIGUSR1 or
> SIGURST2.

Did you mean SIGUSR2?


-- 
ldv

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

Hi,

On Tue, Sep 07, 2021 at 11:34:16AM +0300, Dmitry V. Levin wrote:
> On Tue, Sep 07, 2021 at 02:12:18AM +0200, Mark Wielaard wrote:
> [...]
> > Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle
> > 
> > At startup the debuginfod server does a scan and groom cycle. Make
> > sure to wait for that before triggering a new one with SIGUSR1 or
> > SIGURST2.
> 
> Did you mean SIGUSR2?

Sigh. Yes, that was a typo. Sorry.

It seems this did help with the debuginfod testcases. But we were
still seeing some failures on some of the buildbots because it wasn't
early enough. We really should wait for the initial scan (and possibly
groom) cycle to be done before we make any changes to the scan
directories. Otherwise only part of them might be scanned (or groomed)
at the start (especially because some tests populate the scan dirs and
then remove some files, which aren't supported, again).

The attached (pushed) fixes that hopefully.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Make-sure-to-wait-for-initial-scan-and-groom-c.patch --]
[-- Type: text/x-diff, Size: 8771 bytes --]

From 5ce85e424907d82797f5efe59a8416ec537ebd9e Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Tue, 7 Sep 2021 23:56:42 +0200
Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle
 earlier

At startup the debuginfod server does a scan and groom cycle. Make
sure to wait for that before making any changes to the scan dirs.  And
not just right before triggering a new one with SIGUSR1 for scan or
SIGURS2 for groom.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog                          | 12 ++++++++++++
 tests/run-debuginfod-archive-groom.sh    | 10 +++++++---
 tests/run-debuginfod-archive-rename.sh   | 11 +++++++----
 tests/run-debuginfod-artifact-running.sh | 12 +++++++-----
 tests/run-debuginfod-dlopen.sh           |  7 ++++---
 tests/run-debuginfod-extraction.sh       |  6 ++++--
 tests/run-debuginfod-federation-link.sh  |  3 +--
 7 files changed, 42 insertions(+), 19 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 61080c52..42989d19 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,15 @@
+2021-09-07  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-archive-groom.sh: Wait for initial scan and groom
+	cycle before making any changes to the scan dirs.
+	* run-debuginfod-archive-rename.sh: Likewise.
+	* run-debuginfod-artifact-running.sh: Wait for initial scan cycle
+	before making any changes to the scan dirs.
+	* run-debuginfod-dlopen.sh: Likewise.
+	* run-debuginfod-extraction.sh: Likewise.
+	* run-debuginfod-federation-link.sh: Don't wait twice for the initial
+	scan.
+
 2021-09-06  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-archive-groom.sh: Wait for initial scan and groom
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
index 1e7bd67a..7813ee28 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -46,6 +46,13 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# wait till the initial scan is done before triggering a new one
+# and before dropping new file into the scan dirs
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+# Same for the initial groom cycle, we don't want it to be done
+# half way initializing the file setup
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+
 # Build a non-stripped binary
 echo "int main() { return 0; }" > ${PWD}/F/prog.c
 gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
@@ -58,8 +65,6 @@ if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
-# wait till the initial scan is done before triggering a new one
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
@@ -140,7 +145,6 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
 # debuginfod has forgotten them, but remembers others
 rm -r R/debuginfod-rpms/rhel6/*
 
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 kill -USR2 $PID1  # groom cycle
 ## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
 wait_ready $PORT1 'thread_work_total{role="groom"}' 2
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
index 4fc1b441..5369949b 100755
--- a/tests/run-debuginfod-archive-rename.sh
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -44,13 +44,18 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# wait till the initial scan is done before triggering a new one
+# and before dropping new file into the scan dirs
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+# Same for the initial groom cycle, we don't want it to be done
+# half way initializing the file setup
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+
 cp -rvp ${abs_srcdir}/debuginfod-rpms R
 if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
-# Make sure the initial scan is done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
@@ -65,8 +70,6 @@ SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
 # there are two copies of the same buildid in the index, one for the
 # no-longer-existing file name, and one under the new name.
 
-# Make sure the initial groom cycle has been done
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 # run a groom cycle to force server to drop its fdcache
 kill -USR2 $PID1  # groom cycle
 wait_ready $PORT1 'thread_work_total{role="groom"}' 2
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
index 4eae0200..51fa9c0a 100755
--- a/tests/run-debuginfod-artifact-running.sh
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -42,6 +42,13 @@ env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d
 PID1=$!
 tempfiles vlog$PORT1
 errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# And the initial scan should have been done before moving
+# files under the scan dirs.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 mv prog F
 mv prog.debug F
 tempfiles prog/F
@@ -49,11 +56,6 @@ tempfiles prog/F
 # Be patient when run on a busy machine things might take a bit.
 export DEBUGINFOD_TIMEOUT=10
 
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-# And the initial scan should have been done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-
 kill -USR1 $PID1
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
index 5f33394a..39ee5190 100755
--- a/tests/run-debuginfod-dlopen.sh
+++ b/tests/run-debuginfod-dlopen.sh
@@ -51,6 +51,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# Make sure the initial scan has finished.
+# Before moving files under the scan dirs.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 # We use -t0 and -g0 here to turn off time-based scanning & grooming.
 # For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
 
@@ -71,9 +75,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 mv p+r%o\$g F
 mv p+r%o\$g.debug F
 
-# Make sure the initial scan has finished.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-
 kill -USR1 $PID1
 # Wait till both files are in the index.
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
diff --git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
index 4750f184..06f60e78 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -47,6 +47,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# Make sure the initial scan has finished before copying the new files in
+# We might remove some, which we don't want to be accidentially scanned.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 cp -rvp ${abs_srcdir}/debuginfod-rpms R
 if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
@@ -54,8 +58,6 @@ fi
 
 cp -rvp ${abs_srcdir}/debuginfod-tars Z
 
-# Make sure the initial scan has finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
index 1aef7174..050bcbcf 100755
--- a/tests/run-debuginfod-federation-link.sh
+++ b/tests/run-debuginfod-federation-link.sh
@@ -43,6 +43,7 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 
 wait_ready $PORT1 'ready' 1
+# Make sure initial scan was done
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
@@ -63,8 +64,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 mv prog F
 mv prog.debug F
 
-# Make sure initial scan was done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index.
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-- 
2.32.0

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/813

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/807

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/811

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/807

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/777

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64le while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/762

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/243

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Fri, 2021-09-03 at 10:33 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/813
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)

Gah, sorry. That was my fault.
The last minute change to move stuff into its own file.
Forgot to to add that to EXTRA_DIST.
Hopefully fixed by:

commit 70f36e32cfd83bdaf8a255613acff9ec2b0c0cde
Author: Mark Wielaard <mark@klomp.org>
Date:   Fri Sep 3 13:10:43 2021 +0200

    tests: Add debuginfod-subr.sh to EXTRA_DIST
    
    Signed-off-by: Mark Wielaard <mark@klomp.org>

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 0529f641..9540f738 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,7 @@
+2021-09-03  Mark Wielaard  <mark@klomp.org>
+
+	* debuginfod-subr.sh (EXTRA_DIST): Add debuginfod-subr.sh.
+
 2021-08-28  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-find.sh: Use ":memory:" for the
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 3c302e72..c586422e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -300,7 +300,7 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
 	     testfile21.bz2 testfile21.index.bz2 \
 	     testfile22.bz2 testfile23.bz2 testfile24.bz2 testfile25.bz2 \
 	     testfile26.bz2 testfile27.bz2 \
-	     coverage.sh test-subr.sh test-wrapper.sh \
+	     coverage.sh test-subr.sh test-wrapper.sh debuginfod-subr.sh \
 	     run-readelf-test1.sh run-readelf-test2.sh run-readelf-test3.sh \
 	     run-readelf-test4.sh run-readelf-twofiles.sh \
 	     run-bug1-test.sh testfile28.bz2 testfile28.rdwr.bz2 \

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/803

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Di Chen <dichen@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/807

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Di Chen <dichen@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/239

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Di Chen <dichen@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

Hi,

On Fri, Aug 27, 2021 at 11:19:14PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/803
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64
> 
> Build Reason: <unknown>
> Blamelist: Di Chen <dichen@redhat.com>
> 
> BUILD FAILED: failed test (failure)

This took me a while to figure out because it does not consistently
fail.  It turns out that sharing a database between two debuginfod
instances (especially if they forward queries to each other) is not a
good idea.

Hopefully fixed with the attached patch.

At first I had hoped that using -d :memory: would work. But debuginfod
doesn't really support in memory databases like that. It opens the
database twice in read/write and read-only mode. But :memory: is
special and isn't always a new database, so the read-only copy doesn't
see what is in the read/write copy.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Use-fresh-separate-databases-for-debuginfd-for.patch --]
[-- Type: text/x-diff, Size: 2123 bytes --]

From 17a9b1303e533c13aac6550844bdd68c669091bf Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 28 Aug 2021 15:54:18 +0200
Subject: [PATCH] tests: Use fresh separate databases for debuginfd
 forwarded-ttl-limit

Sharing the database between the two debuginfod instances that forward
queries to each other causes issues. Make both debuginfod instances
use a new fresh database.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog              | 5 +++++
 tests/run-debuginfod-find.sh | 7 +++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 29c48b97..cbd1c227 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2021-08-28  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-find.sh: Use clean, separate databases for
+	forwarded-ttl-limit tests.
+
 2021-08-20  Di Chen  <dichen@redhat.com>
 
 	* run-debuginfod-find.sh: Add test for X-Forwarded-For hops limit.
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index 7e12dd7f..5d38d625 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -824,10 +824,13 @@ done
 tempfiles vlog$PORT4 vlog$PORT5
 errfiles vlog$PORT4 vlog$PORT5
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
+# Give each debuginfd its own clean database.
+tempfiles db.$PORT4.sql db.$PORT5.sql
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
 PID5=$!
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
 PID6=$!
 
 wait_ready $PORT4 'ready' 1
-- 
2.32.0

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Frank Ch. Eigler]

Hi -

> This took me a while to figure out because it does not consistently
> fail.  It turns out that sharing a database between two debuginfod
> instances (especially if they forward queries to each other) is not a
> good idea.

That's correct.  Each test case invocation of debuginfod should be run
with its own -d $database.

> Hopefully fixed with the attached patch.
> At first I had hoped that using -d :memory: would work. But debuginfod
> doesn't really support in memory databases like that. [...]

Yes, that functionality was broken at the point you identifed.  We could
bring it back by hacking debuginfod's command line parser to map
    -d :memory:
to  -d file::memory:?cache=shared

(https://sqlite.org/inmemorydb.html)

- FChE

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/807

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/801

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/805

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/801

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/771

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64le while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/756

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/754

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-armhf while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/596

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/237

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Saleem Abdulrasool <abdulras@google.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Fri, Aug 27, 2021 at 09:20:22PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/807
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Saleem Abdulrasool <abdulras@google.com>
> 
> BUILD FAILED: failed compile (failure)

This wasn't Saleem's fault. I didn't apply his patch correctly and
forgot to include a file. Fixed by:

commit 610623458b7e98ed3e912e4b7ca8050f6ce4c698
Author: Mark Wielaard <mark@klomp.org>
Date:   Fri Aug 27 18:47:30 2021 +0200

    Add lib/error.c
    
    This new file was supposed to be part of 4d6dd0e5a "lib: avoid potential
    problems with `-fno-common`".
    
    Signed-off-by: Mark Wielaard <mark@klomp.org>

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/799

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Alice Zhang <alizhang@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/793

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Alice Zhang <alizhang@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/797

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Alice Zhang <alizhang@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/793

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Alice Zhang <alizhang@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/763

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Alice Zhang <alizhang@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Wed, Aug 11, 2021 at 11:16:18PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/799

Just for the record, this failure was fixed by:

commit f3466e18337681d2159bb591aaee6993e6df4fee
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Wed Aug 11 19:32:29 2021 -0400

    debuginfod-doc: PR27950: make distcheck happy
    
    The debuginfod-client-config.7 shouldn't be included twice in
    notrans_dist_man7_MANS.
    
    Signed-off-by: Frank Ch. Eigler <fche@redhat.com>

diff --git a/doc/Makefile.am b/doc/Makefile.am
index 7979be4d..32d1a2b8 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -24,7 +24,9 @@ notrans_dist_man8_MANS=
 notrans_dist_man1_MANS=
 
 if DEBUGINFOD
+if !LIBDEBUGINFOD
 notrans_dist_man7_MANS += debuginfod-client-config.7
+endif
 notrans_dist_man8_MANS += debuginfod.8
 endif
 
After which all builds have been green again.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/796

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, 2021-08-04 at 15:13 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/796
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Noah Sanci <nsanci@redhat.com>
> 
> BUILD FAILED: failed test (failure)

Not sure why this happened, the actual "failure" was:

PASS: run-large-elf-file.sh

command timed out: 1200 seconds without output running ['make',
'check', '-j4'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1365.281001

Normally run-large-elf-file.sh is skipped because it eats up all the
memory. Maybe that created a swap storm?

A rebuild was all green.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/786

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/790

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/785

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64le while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/740

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-ppc64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/738

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/221

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Noah Sanci <nsanci@redhat.com>

BUILD FAILED: failed update (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Thu, 2021-07-22 at 14:24 +0000, buildbot@builder.wildebeest.org
wrote:
> BUILD FAILED: failed update (failure)

So this was some issue at sourceware, the git update broke for some
builders. I have restarted the builds on the buildbot and things seems
to be green again (s390x, amd64, i386 are still running though).

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/790

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-centos-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/784

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Andrei Homescu <ah@immunant.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/778

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Andrei Homescu <ah@immunant.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/783

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Andrei Homescu <ah@immunant.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-i386 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/778

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Andrei Homescu <ah@immunant.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-s390x while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/749

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Andrei Homescu <ah@immunant.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Thu, 2021-07-08 at 09:53 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a new failure on builder elfutils-centos-
> x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/784
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Andrei Homescu <ah@immunant.com>
> 
> BUILD FAILED: failed test (failure)

Oops. Thanks buildbot.
Typo in EXTRA_DIST that only shows up when running make distcheck.

Fixed with the attached commit.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Fix-EXTRA_DIST-typo-in-testfile-largealign.o.b.patch --]
[-- Type: text/x-patch, Size: 1284 bytes --]

From cdcd7dc3d20a002abe1ce318f9b9d0895eee1810 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Thu, 8 Jul 2021 11:44:26 +0200
Subject: [PATCH] tests: Fix EXTRA_DIST typo in testfile-largealign.o.bz2

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog   | 5 +++++
 tests/Makefile.am | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 28aaf85e..7b493c99 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2021-07-08  Mark Wielaard  <mark@klomp.org>
+
+	* Makefile.am (EXTRA_DIST): Fix typo testfile-largealign.bz2 was
+	was missing .o.
+
 2021-06-09  Andrei Homescu  <ah@immunant.com>
 
 	* testfile-largealign.o.bz2: New test file.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 8ac0d2e6..429649f4 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -512,7 +512,7 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
 	     run-getphdrnum.sh testfile-phdrs.elf.bz2 \
 	     run-test-includes.sh run-low_high_pc-dw-form-indirect.sh \
 	     run-readelf-dw-form-indirect.sh testfile-dw-form-indirect.bz2 \
-	     testfile-largealign.bz2 run-strip-largealign.sh
+	     testfile-largealign.o.bz2 run-strip-largealign.sh
 
 
 if USE_VALGRIND
-- 
2.18.4

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/770

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Omar Sandoval <osandov@fb.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Tue, Jun 15, 2021 at 09:29:46PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/770
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Omar Sandoval <osandov@fb.com>
> 
> BUILD FAILED: failed test (failure)

This was a test to see if ccache was installed correctly on the worker.
It seems it is. But run-find-debuginfo.sh failed for (I think) unrelated reasons.

Sorry,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Frank Ch. Eigler]

Hi -

> [...] But run-find-debuginfo.sh failed for (I think) unrelated reasons.

Any objection to me firing in some speculative tests/run-debuginfod*
patches into the buildbot to evoke more usefully verbose failures?

- FChE

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi Frank,

On Tue, Jun 15, 2021 at 05:53:18PM -0400, Frank Ch. Eigler wrote:
> > [...] But run-find-debuginfo.sh failed for (I think) unrelated reasons.
> 
> Any objection to me firing in some speculative tests/run-debuginfod*
> patches into the buildbot to evoke more usefully verbose failures?

If you think it helps diagnose these spurious failure please do.  It
might also help to split this test into smaller ones to make sure the
tests just test one specific thing.

Thanks,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Frank Ch. Eigler]

Hi -

> If you think it helps diagnose these spurious failure please do.  

(Shall I try to get a wildebeest buildbot admin login to try to
rebuild repeatedly, till it hits these heisenbugs?)

> It might also help to split this test into smaller ones to make sure
> the tests just test one specific thing.

Yeah, thinking about it.  It just seems that a lot of the
initialization between tests would be identical, so them being
cumulative vs. individual eliminates duplication.  In case of a test
failure, the whole suite fails anyway, so cumulative vs. individual
doesn't help there.

- FChE

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

H Frank,

On Tue, 2021-06-15 at 19:00 -0400, Frank Ch. Eigler wrote:
> (Shall I try to get a wildebeest buildbot admin login to try to
> rebuild repeatedly, till it hits these heisenbugs?)

Yes, good idea. I'll sent you the credentials so you can hit the
rebuild button as often as you like.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/202

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Sergei Trofimovich <slyfox@gentoo.org>

BUILD FAILED: failed configure (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, Jun 12, 2021 at 11:39:12PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/45/builds/202
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-arm64
> 
> Build Reason: <unknown>
> Blamelist: Sergei Trofimovich <slyfox@gentoo.org>
> 
> BUILD FAILED: failed configure (failure)

Sorry, that was my fault, I was experimenting with ccache to speed up
builds, but misconfigured things. It should be setup correctly now.

Apologies,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/748

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/702

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Tue, Apr 13, 2021 at 04:54:13PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/11/builds/702
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64le

This was a different issue:

FAIL: run-backtrace-native-core.sh
==================================
/usr/bin/coredumpctl
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
           PID: 10643 (backtrace-child)
           UID: 1002 (mjw)
           GID: 1002 (mjw)
        Signal: 6 (ABRT)
     Timestamp: Tue 2021-04-13 12:59:04 UTC (2s ago)
  Command Line: /home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child --gencore
    Executable: /home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child
 Control Group: /user.slice/user-1002.slice/session-2.scope
          Unit: session-2.scope
         Slice: user-1002.slice
       Session: 2
     Owner UID: 1002 (mjw)
       Boot ID: 4684256e966845baad90ffbef2d3c976
    Machine ID: fa20d94f66194772a93b94464bd75866
      Hostname: rh-power-vm60.fit.vutbr.cz
       Storage: /var/lib/systemd/coredump/core.backtrace-child.1002.4684256e966845baad90ffbef2d3c976.10643.1618318744000000.lz4
       Message: Process 10643 (backtrace-child) of user 1002 dumped core.
                
                Stack trace of thread 10644:
                #0  0x00007fffadc28d48 raise (libpthread.so.0)
                #1  0x000000012d5e14a4 n/a (/home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child)
                #2  0x000000012d5e15cc n/a (/home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child)
                #3  0x000000012d5e161c n/a (/home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child)
                #4  0x000000012d5e1648 n/a (/home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace-child)
                #5  0x00007fffadc18c10 start_thread (libpthread.so.0)
                #6  0x00007fffadb2d8a8 __clone (libc.so.6)
0x7fffae620000	0x7fffae630000	linux-vdso64.so.1
0x7fffae640000	0x7fffae681108	ld64.so.2
0x7fffad590000	0x7fffad5c0428	libgcc_s.so.1
0x7fffad5d0000	0x7fffad6f0128	libm.so.6
0x7fffad700000	0x7fffad9955f8	libstdc++.so.6
0x7fffad9a0000	0x7fffad9c0320	librt.so.1
0x7fffad9d0000	0x7fffad9f0108	libdl.so.2
0x7fffada00000	0x7fffadc05378	libc.so.6
0x7fffadc10000	0x7fffadc54520	libpthread.so.0
0x7fffadc60000	0x7fffae5f51c8	libubsan.so.1
0x12d5e0000	0x12d6001c0	backtrace-child
TID 10644:
# 0 0x7fffadc28d48    	raise
# 1 0x12d5e14a4 - 1	sigusr2
# 2 0x12d5e15cc - 1	stdarg
# 3 0x12d5e161c - 1	backtracegen
# 4 0x12d5e1648 - 1	start
# 5 0x7fffadc18c10 - 1	start_thread
# 6 0x7fffadb2d8a8 - 1	__clone
/home/mjw/wildebeest/buildbot/elfutils-fedora-ppc64le/build/tests/backtrace: dwfl_thread_getframes: address out of range
backtrace: backtrace.c:81: callback_verify: Assertion `seen_main' failed.
./test-subr.sh: line 84: 10904 Aborted                 (core dumped) LD_LIBRARY_PATH="${built_library_path}${LD_LIBRARY_PATH:+:}$LD_LIBRARY_PATH" $VALGRIND_CMD "$@"
backtrace-child-core.10643: no main
rmdir: failed to remove 'test-10634': Directory not empty
FAIL run-backtrace-native-core.sh (exit status: 1)

It disappeared on a rebuild...

It looks like in the failure case the child thread was unwound
correctly, but the main thread couldn't. It is unclear to be if this
was because of a bug in the unwinder or because systemd left us a bad
core file.

The corresponding change (commit 879513ab - nm: Fix file descriptor
leak on dwfl_begin failure.) really couldn't have caused this IMHO. So
it is a bit of a mystery.

Sigh,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/747

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Tue, Apr 13, 2021 at 02:29:13PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/747

Sadly there is still the occasional FAIL in the run-debuginfod-find.sh
testcase.  It only happens once in a couple of builds, it didn't
happen on the next build on the exact same buildbot worker, but I
still haven't figured out why or when.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/714

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, Mar 31, 2021 at 11:29:10AM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/714
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x
> 
> Build Reason: <unknown>
> Blamelist: Frank Ch. Eigler <fche@redhat.com>

This cannot be blamed on Frank. Sorry.  It looks like the build
(compile) timed out.  No idea if that was because the machine is
overloaded or something else. I'll try to rebuild it.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/742

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/740

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Piotr Drąg <piotrdrag@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/740

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Piotr Drąg <piotrdrag@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, Mar 06, 2021 at 02:45:23PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/740
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64
> 
> Build Reason: <unknown>
> Blamelist: Piotr Drąg <piotrdrag@gmail.com>
> 
> BUILD FAILED: failed test (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/4/builds/740
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-i386

Both failures seem unrelated to the Polish translation just
added. Both are in the run-debuginfod-find.sh testcase. So I guess
this testcase is still unstable :{

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/734

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/736

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/733

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/700

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/693

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Mon, 2021-03-01 at 19:56 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/693
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x
> 
> Build Reason: <unknown>
> Blamelist: Timm Bäder <tbaeder@redhat.com>
> 
> BUILD FAILED: failed test (failure)

I don't think it was Timm's patch. It seems the run-debuginfod-find.sh
test is still unstable. But now it only happens on s390x. And there are
at least 5 good builds between the failures.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/724

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/155

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/685

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Alexander Miller via Elfutils-devel <elfutils-devel@sourceware.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/713

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/667

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Érico Rolim <erico.erc@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Tue, 2021-02-09 at 02:26 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/11/builds/667
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64le
> 
> Build Reason: <unknown>
> Blamelist: Érico Rolim <erico.erc@gmail.com>
> 
> BUILD FAILED: failed test (failure)

Well, that is unfortunate. This couldn't really have been caused by
your patch. But sadly the run-debuginfod-find.sh seems a little
unstable :{ Full details here:
https://builder.wildebeest.org/buildbot/#/builders/11/builds/667/steps/8/logs/test-suite_log

Still looking at what is going wrong.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Tue, Feb 09, 2021 at 10:05:05AM +0100, Mark Wielaard wrote:
> On Tue, 2021-02-09 at 02:26 +0000, buildbot@builder.wildebeest.org
> wrote:
> > The Buildbot has detected a failed build on builder whole buildset
> > while building elfutils.
> > Full details are available at:
> >     https://builder.wildebeest.org/buildbot/#builders/11/builds/667
> > 
> > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > 
> > Worker for this Build: fedora-ppc64le
> > 
> > Build Reason: <unknown>
> > Blamelist: Érico Rolim <erico.erc@gmail.com>
> > 
> > BUILD FAILED: failed test (failure)
> 
> Well, that is unfortunate. This couldn't really have been caused by
> your patch. But sadly the run-debuginfod-find.sh seems a little
> unstable :{ Full details here:
> https://builder.wildebeest.org/buildbot/#/builders/11/builds/667/steps/8/logs/test-suite_log
> 
> Still looking at what is going wrong.

Interestingly enough the same test case now succeeded on
fedora-ppc64le, but failed on debian-i386...

They failed in the same way:
  metric groom{statistic="file d/e"} never changed to 3 on port 9985

Still don't know why though :{

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Fri, 2021-02-12 at 16:58 +0100, Mark Wielaard wrote:
> Hi,
> 
> On Tue, Feb 09, 2021 at 10:05:05AM +0100, Mark Wielaard wrote:
> > On Tue, 2021-02-09 at 02:26 +0000, buildbot@builder.wildebeest.org
> > wrote:
> > > The Buildbot has detected a failed build on builder whole buildset
> > > while building elfutils.
> > > Full details are available at:
> > >     https://builder.wildebeest.org/buildbot/#builders/11/builds/667
> > > 
> > > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > > 
> > > Worker for this Build: fedora-ppc64le
> > > 
> > > Build Reason: <unknown>
> > > Blamelist: Érico Rolim <erico.erc@gmail.com>
> > > 
> > > BUILD FAILED: failed test (failure)
> > 
> > Well, that is unfortunate. This couldn't really have been caused by
> > your patch. But sadly the run-debuginfod-find.sh seems a little
> > unstable :{ Full details here:
> > https://builder.wildebeest.org/buildbot/#/builders/11/builds/667/steps/8/logs/test-suite_log
> > 
> > Still looking at what is going wrong.
> 
> Interestingly enough the same test case now succeeded on
> fedora-ppc64le, but failed on debian-i386...
> 
> They failed in the same way:
>   metric groom{statistic="file d/e"} never changed to 3 on port 9985
> 
> Still don't know why though :{

And now debian-i386 and fedora-ppc64 both succeeded, but fedora-s390x
failed! Gah. At least the error is again the same.

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/634

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed 'autoreconf -f ...' (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/633

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed 'autoreconf -f ...' (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

Don't worry...

On Wed, Dec 16, 2020 at 11:54:16PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/11/builds/634
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64le
> 
> Build Reason: <unknown>
> Blamelist: Dmitry V. Levin <ldv@altlinux.org>
> 
> BUILD FAILED: failed 'autoreconf -f ...' (failure)
> 
> Sincerely,
>  -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/12/builds/633
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64
> 
> Build Reason: <unknown>
> Blamelist: Dmitry V. Levin <ldv@altlinux.org>
> 
> BUILD FAILED: failed 'autoreconf -f ...' (failure)

The ppc64 and ppc64le builders didn't have autopoint installed
(gettext-devel) they have now and newer builds have secceeded.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/611

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, Dec 12, 2020 at 05:11:48PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/11/builds/611
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64le
> 
> Build Reason: <unknown>
> Blamelist: Dmitry V. Levin <ldv@altlinux.org>
> 
> BUILD FAILED: failed test (failure)

That is odd. The patch was just a spelling fix. It only failed on
ppc64le. I cannot replicate it. And there have been 12 successful
builds after it.

Lets assume it was a random failure and nothing serious.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/640

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/436

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Sat, 2020-11-28 at 04:15 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/4/builds/640
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-i386
> 
> The Buildbot has detected a failed build on builder
> whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/15/builds/436
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-armhf

That is interesting, it only fails on the two 32bit systems.
The failure is about the specific error message returned.

FAIL: run-dwflsyms.sh
=====================
--- dwflsyms.out	2020-11-28 01:17:44.130295202 +0000
+++ -	2020-11-28 01:17:44.140327188 +0000
@@ -10,7 +10,7 @@
    9: NOTYPE	GLOBAL	__kernel_sigtramp_rt64 (12) 0xfffb1af0418
   10: NOTYPE	GLOBAL	__kernel_clock_gettime (152) 0xfffb1af0494
   11: NOTYPE	GLOBAL	__kernel_get_syscall_map (44) 0xfffb1af05f4
-ld64.so.1: No symbol table found
+ld64.so.1: Callback returned failure
    0: NOTYPE	LOCAL	 (0) 0
    1: SECTION	LOCAL	 (0) 0x461b0190
    2: SECTION	LOCAL	 (0) 0x461b01a4
FAIL run-dwflsyms.sh (exit status: 1)

So there is some subtle difference in the behavior of the
'segment_report_module: Inline consider_notes() into only caller'
patch.

I haven't spotted it yet, but I suspect some 'return' from the original
function got mistranslated as a continue, break or goto out in the
inlined variant.

The specific testcase that fails is:
testrun_compare ${abs_builddir}/dwflsyms -e testfile66 --core=testfile66.core

Which is a big endian ppc64 executable and core file.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, 2020-11-28 at 14:41 +0100, Mark Wielaard wrote:
> That is interesting, it only fails on the two 32bit systems.
> The failure is about the specific error message returned.
> 
> FAIL: run-dwflsyms.sh
> =====================
> --- dwflsyms.out	2020-11-28 01:17:44.130295202 +0000
> +++ -	2020-11-28 01:17:44.140327188 +0000
> @@ -10,7 +10,7 @@
>     9: NOTYPE	GLOBAL	__kernel_sigtramp_rt64 (12) 0xfffb1af0418
>    10: NOTYPE	GLOBAL	__kernel_clock_gettime (152) 0xfffb1af0494
>    11: NOTYPE	GLOBAL	__kernel_get_syscall_map (44) 0xfffb1af05f4
> -ld64.so.1: No symbol table found
> +ld64.so.1: Callback returned failure
>     0: NOTYPE	LOCAL	 (0) 0
>     1: SECTION	LOCAL	 (0) 0x461b0190
>     2: SECTION	LOCAL	 (0) 0x461b01a4
> FAIL run-dwflsyms.sh (exit status: 1)
> 
> So there is some subtle difference in the behavior of the
> 'segment_report_module: Inline consider_notes() into only caller'
> patch.
> 
> I haven't spotted it yet, but I suspect some 'return' from the original
> function got mistranslated as a continue, break or goto out in the
> inlined variant.
> 
> The specific testcase that fails is:
> testrun_compare ${abs_builddir}/dwflsyms -e testfile66 --core=testfile66.core
> 
> Which is a big endian ppc64 executable and core file.

The issue can be replicated on x86_64 with:

$ CXX="g++ -m32" CC="gcc -m32" ~/src/elfutils/configure --enable-maintainer-mode
$ make -j4
$ make check TESTS=run-dwflsyms.sh

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

Hi,

On Sat, Nov 28, 2020 at 04:12:25PM +0100, Mark Wielaard wrote:
> On Sat, 2020-11-28 at 14:41 +0100, Mark Wielaard wrote:
> > That is interesting, it only fails on the two 32bit systems.
> > The failure is about the specific error message returned.
> > 
> > FAIL: run-dwflsyms.sh
> > =====================
> > --- dwflsyms.out	2020-11-28 01:17:44.130295202 +0000
> > +++ -	2020-11-28 01:17:44.140327188 +0000
> > @@ -10,7 +10,7 @@
> >     9: NOTYPE	GLOBAL	__kernel_sigtramp_rt64 (12) 0xfffb1af0418
> >    10: NOTYPE	GLOBAL	__kernel_clock_gettime (152) 0xfffb1af0494
> >    11: NOTYPE	GLOBAL	__kernel_get_syscall_map (44) 0xfffb1af05f4
> > -ld64.so.1: No symbol table found
> > +ld64.so.1: Callback returned failure
> >     0: NOTYPE	LOCAL	 (0) 0
> >     1: SECTION	LOCAL	 (0) 0x461b0190
> >     2: SECTION	LOCAL	 (0) 0x461b01a4
> > FAIL run-dwflsyms.sh (exit status: 1)
> > 
> > So there is some subtle difference in the behavior of the
> > 'segment_report_module: Inline consider_notes() into only caller'
> > patch.
> > 
> > I haven't spotted it yet, but I suspect some 'return' from the original
> > function got mistranslated as a continue, break or goto out in the
> > inlined variant.
> > 
> > The specific testcase that fails is:
> > testrun_compare ${abs_builddir}/dwflsyms -e testfile66 --core=testfile66.core
> > 
> > Which is a big endian ppc64 executable and core file.
> 
> The issue can be replicated on x86_64 with:
> 
> $ CXX="g++ -m32" CC="gcc -m32" ~/src/elfutils/configure --enable-maintainer-mode
> $ make -j4
> $ make check TESTS=run-dwflsyms.sh

Found it. There was a small change to calculate note_vaddr (so it
wouldn't clash with the local vaddr) as:

  const size_t note_vaddr = start + offset;

But size_t is only 32bit on the failing systems, which is obviously
not enough for analyzing addresses from a 64bit core file. The fix is
simply to use GElf_Addr instead. Pushed the attached to fix it.

Cheers,

Mark

[-- Attachment #2: 0001-libdwfl-Use-64bit-GElf_Addr-instead-of-size_t-to-cal.patch --]
[-- Type: text/x-diff, Size: 1618 bytes --]

From 609290a61d4f900c65b7e0e273981022a826e4c0 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sun, 29 Nov 2020 01:57:53 +0100
Subject: [PATCH] libdwfl: Use 64bit GElf_Addr instead of size_t to calculate
 address.

size_t is too small on 32 bit systems to analyze a 64 bit core file.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 libdwfl/ChangeLog                    | 5 +++++
 libdwfl/dwfl_segment_report_module.c | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/libdwfl/ChangeLog b/libdwfl/ChangeLog
index a5cffc49..67a4d743 100644
--- a/libdwfl/ChangeLog
+++ b/libdwfl/ChangeLog
@@ -1,3 +1,8 @@
+2020-11-28  Mark Wielaard  <mark@klomp.org>
+
+	* dwfl_segment_report_module.c (dwfl_segment_report_module):
+	Use GElf_Addr to calculate note_vaddr instead of size_t.
+
 2020-11-26  Timm Bäder  <tbaeder@redhat.com>
 
 	* dwfl_segment_report_module.c (dwfl_segment_report_module):
diff --git a/libdwfl/dwfl_segment_report_module.c b/libdwfl/dwfl_segment_report_module.c
index 8d99e3bb..ee9cfa2e 100644
--- a/libdwfl/dwfl_segment_report_module.c
+++ b/libdwfl/dwfl_segment_report_module.c
@@ -501,7 +501,7 @@ dwfl_segment_report_module (Dwfl *dwfl, int ndx, const char *name,
 
               /* We calculate from the p_offset of the note segment,
                because we don't yet know the bias for its p_vaddr.  */
-              const size_t note_vaddr = start + offset;
+              const GElf_Addr note_vaddr = start + offset;
               void *data;
               size_t data_size;
               if (read_portion (&read_state, &data, &data_size,
-- 
2.18.4

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/630

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Mon, 2020-11-23 at 17:50 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/630
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64
> 
> Build Reason: <unknown>
> Blamelist: Mark Wielaard <mark@klomp.org>
> 
> BUILD FAILED: failed test (failure)

Interestingly only one buildbot worker found this issue, but I could
replicate it under valgrind locally. And indeed, valgrind is right. We
forgot to initialize the new handler_data errbuf field.

Fixed as attached.

Cheers,

Mark

[-- Attachment #2: 0001-debuginfod-client-Initialize-struct-handle_data-errb.patch --]
[-- Type: text/x-patch, Size: 1230 bytes --]

From 6b82ac67d8a71c14f64fe4932ca7fe822ff75231 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 23 Nov 2020 17:52:02 +0100
Subject: [PATCH] debuginfod-client: Initialize struct handle_data errbuf to
 the empty string.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 debuginfod/ChangeLog           | 5 +++++
 debuginfod/debuginfod-client.c | 1 +
 2 files changed, 6 insertions(+)

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index bd4dbf40..cd009fd2 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,8 @@
+2020-11-23  Mark Wielaard  <mark@klomp.org>
+
+	* debuginfod-client.c (debuginfod_query_server): Initialize
+	struct handle_data errbuf to the empty string.
+
 2020-11-11  Mark Wielaard  <mark@klomp.org>
 
 	* debuginfod-client.c (debuginfod_set_verbose_fd): New function.
diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
index 2bf1543a..a99f3c14 100644
--- a/debuginfod/debuginfod-client.c
+++ b/debuginfod/debuginfod-client.c
@@ -756,6 +756,7 @@ debuginfod_query_server (debuginfod_client *c,
     {
       data[i].handle = NULL;
       data[i].fd = -1;
+      data[i].errbuf[0] = '\0';
     }
 
   char *strtok_saveptr;
-- 
2.18.4

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/591

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Timm Bäder <tbaeder@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Thu, 2020-11-19 at 13:26 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/591
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x
> 
> Build Reason: <unknown>
> Blamelist: Timm Bäder <tbaeder@redhat.com>
> 
> BUILD FAILED: failed test (failure)

No Worries Timm, this isn't on you. The build on that particular
machine just timed out:

   command timed out: 1200 seconds without output running [b'make',
   b'distcheck', b'-j2'], attempting to kill

I'll talk to the machine owner to see if we aren't overloading the
machine. The last build took 7 hours (!). Currently we are doing a full
make distcheck which runs the test suite 3 times, twice under valgrind.
But given that this our only big-endian tester I like to keep it.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/587

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Tue, 2020-11-17 at 15:09 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/587
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x
> 

I couldn't replicate it and a rebuild is all green:
https://builder.wildebeest.org/buildbot/#/builders/10/builds/588

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/600

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/600

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/38

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Hi,

On Sat, 2020-10-31 at 02:12 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/600
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64
> 
> Build Reason: <unknown>
> Blamelist: Mark Wielaard <mark@klomp.org>
> 
> BUILD FAILED: failed test (failure)
> [...]
> Worker for this Build: debian-i386
> [...]
> Worker for this Build: debian-arm64

Debian, debian, debian.
Debian uses dash as /bin/sh which is pretty strict about syntax. It
didn't like the== in  the test for strings in the test-wrapper.sh.

Fixed as attached.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Remove-bashism-from-test-wrapper.sh-bin-sh-scr.patch --]
[-- Type: text/x-patch, Size: 1349 bytes --]

From 92692b817028499111a5446cf1bc8369d0a3b408 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 31 Oct 2020 11:19:23 +0100
Subject: [PATCH] tests: Remove bashism from test-wrapper.sh /bin/sh script.

Debian uses dash as /bin/sh which is pretty strict about syntax. It
didn't like the == in  the test for strings in the test-wrapper.sh.
Replace by single =.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog       | 4 ++++
 tests/test-wrapper.sh | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index b84f2af0..17c6ba2d 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,7 @@
+2020-10-31  Mark Wielaard  <mark@klomp.org>
+
+	* test-wrapper.sh: Use =, not == for string compare.
+
 2020-10-29  Mark Wielaard  <mark@klomp.org>
 
 	* test-wrapper.sh: Determine whether the test is a script or not
diff --git a/tests/test-wrapper.sh b/tests/test-wrapper.sh
index db16c591..c5d3f159 100755
--- a/tests/test-wrapper.sh
+++ b/tests/test-wrapper.sh
@@ -66,7 +66,7 @@ fi
 
 # When it is a run-*.sh script the VALGRIND_CMD will be passed on
 # otherwise we'll need to run the binary explicitly under valgrind.
-if [ "x$is_shell_script" == "xyes" ]; then
+if [ "x$is_shell_script" = xyes ]; then
   exec "$@"
 else
   exec $VALGRIND_CMD "$@"
-- 
2.18.4

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/603

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/597

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/600

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/597

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/568

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/553

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/552

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/394

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/45/builds/35

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-arm64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/549

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Fri, Oct 30, 2020 at 01:51:35AM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/12/builds/549
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-ppc64
> 
> BUILD FAILED: failed test (failure)

Hmmm, this is odd.

FAIL: run-debuginfod-find.sh

The full log is at:
https://builder.wildebeest.org/buildbot/#/builders/12/builds/549/steps/8/logs/test-suite_log

But I don't fully understand what went wrong.
Also the commit that triggered this didn't change any code.
It only added a new testcase.

I did a new build, which did succeed:
https://builder.wildebeest.org/buildbot/#/builders/12/builds/550

So this must be a timing issue or something non-deterministic.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/589

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/583

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/586

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/583

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/554

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Thu, Oct 08, 2020 at 04:45:07PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/589

This was make distcheck failing make uninstall.
Frank already fixed it:

commit fee123fb771f46274dfc8260c00f5d3061de2cd2
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Thu Oct 8 11:56:15 2020 -0400

    PR25461: uninstall /etc/profile.d files for distuninstallcheck
    
    No one uses "make uninstall" except the buildbot.  Let's make it
    happy too.
    
    Signed-off-by: Frank Ch. Eigler <fche@redhat.com>

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/16/builds/343

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-aarch64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Fri, 2020-06-19 at 23:20 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/16/builds/343
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-aarch64

That was a compiler crash triggered by a hardware/memory issue:
Internal error: synchronous parity or ECC error: 96000018 [#1] SMP

A reboot and a rebuild made the build pass.
But we probably have to get replacement hardware/memory :{

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/546

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Thu, Jun 11, 2020 at 04:00:38PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/546
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

This is a failure in run-debuginfod-find.sh where an extra groom is
not happening when expected.  I tried to replicate, but
failed. Rebuilding the task on the same builder makes it PASS. This
doesn't fail on any other buildbot worker. I suspect it is a timing
issue with a signal not arriving when expected. But I am not sure. I
don't think this should hold up the release, given that is is probably
just a testsuite quirk.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/503

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/498

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/501

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/498

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/469

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/294

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/16/builds/291

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-aarch64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/495

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, 2020-03-25 at 23:48 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/495
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64

Well, this is odd. This is:

FAIL: run-backtrace-native-core-biarch.sh
=========================================

0xf7f6c000	0xf7f6e000	linux-gate.so.1
0xf7f6e000	0xf7f97948	ld-linux.so.2
0xf7d56000	0xf7f33780	libc.so.6
0xf7f34000	0xf7f542b8	libpthread.so.0
0x565a4000	0x565a8054	backtrace-child-biarch
TID 5133:
# 0 0xf7f6d069    	__kernel_vsyscall
# 1 0xf7f469b2 - 1	raise
# 2 0x565a548a - 1	sigusr2
# 3 0x565a5564 - 1	stdarg
# 4 0x565a557a - 1	backtracegen
# 5 0x565a5588 - 1	start
# 6 0xf7f3afd2 - 1	start_thread
# 7 0xf7e506d6 - 1	__clone
TID 5132:
# 0 0xf7e506c4    	__clone
/srv/buildbot/worker/elfutils-debian-amd64/build/tests/backtrace:
dwfl_thread_getframes: no matching address range
backtrace: backtrace.c:81: callback_verify: Assertion `seen_main'
failed.
./test-subr.sh: line 84:  5136
Aborted                 LD_LIBRARY_PATH="${built_library_path}${LD_LIBR
ARY_PATH:+:}$LD_LIBRARY_PATH" $VALGRIND_CMD "$@"
backtrace-child-biarch-core.5132: no main
rmdir: failed to remove 'test-5124': Directory not empty
FAIL run-backtrace-native-core-biarch.sh (exit status: 1)

But I cannot replicate it. And it was caused by a change to the README
file. Rebuilding on the same buildbot worker also made it PASS... Sigh.

Don't know what to do about it, except watch whether it happens again
or not.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/492

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/288

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Mon, 2020-03-23 at 13:38 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/4/builds/492
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-i386
> [...]
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/15/builds/288
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-armhf

Well, that is odd. Both are 32bit arches. The failure is the same in
run-lfs-symbols.sh

   FAIL: run-lfs-symbols.sh
   ========================
   ./testfile-nolfs contains non-lfs symbols: fopen
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_i386.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_sh.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_x86_64.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_ia64.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_alpha.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_arm.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_aarch64.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_sparc.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_ppc.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_ppc64.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_s390.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_tilegx.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_m68k.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_bpf.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_riscv.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot open
   '/srv/buildbot/worker/elfutils-debian-
   i386/build/backends/libebl_csky.so': No such file or directory
   /srv/buildbot/worker/elfutils-debian-i386/build/tests/getphdrnum
   contains non-lfs symbols: open
   FAIL run-lfs-symbols.sh (exit status: 1)

The backends shared libraries don't exist anymore and should be removed
from the testcase.

But I don't immediately see what/where I forgot which flags to make the
new getphdrnum check program picks up the wrong "open".

Investigating...

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 4084 bytes --]

Hi,

On Mon, 2020-03-23 at 15:08 +0100, Mark Wielaard wrote:
> Well, that is odd. Both are 32bit arches. The failure is the same in
> run-lfs-symbols.sh
> 
>    FAIL: run-lfs-symbols.sh
>    ========================
>    ./testfile-nolfs contains non-lfs symbols: fopen
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_i386.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_sh.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_x86_64.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_ia64.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_alpha.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_arm.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_aarch64.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_sparc.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_ppc.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_ppc64.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_s390.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_tilegx.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_m68k.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_bpf.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_riscv.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/src/nm: cannot
> open
>    '/srv/buildbot/worker/elfutils-debian-
>    i386/build/backends/libebl_csky.so': No such file or directory
>    /srv/buildbot/worker/elfutils-debian-i386/build/tests/getphdrnum
>    contains non-lfs symbols: open
>    FAIL run-lfs-symbols.sh (exit status: 1)
> 
> The backends shared libraries don't exist anymore and should be
> removed
> from the testcase.

Which the attached patch does. It also explicitly fails if a file
doesn't exist and 

> But I don't immediately see what/where I forgot which flags to make
> the new getphdrnum check program picks up the wrong "open".

It simply was a missing #include "config.h" which defines
_FILE_OFFSET_BITS without which you don't get lfs symbols... doh.

Both issues, plus some extra logging in run-lfs-symbols.sh fixed by the
attached patch which I just pushed.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Fix-getphdrnum-and-run-lfs-symbols.sh-testcase.patch --]
[-- Type: text/x-patch, Size: 2939 bytes --]

From 7db645ba72f5711174dbf712003b9600009e5ea7 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 23 Mar 2020 23:57:51 +0100
Subject: [PATCH] tests: Fix getphdrnum and run-lfs-symbols.sh testcase.

getphdrnum.c didn't include config.h which is why run-lfs-symbols.sh
flagged it for containing bad (non-lfs) symbols.

run-lfs-symbols.sh was still checking the libebl modules, which we
don't create anymore. But it didn't fail the test for non-existing
tests. Add some extra logging and explicitly check files exist.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog          |  6 ++++++
 tests/getphdrnum.c       |  2 ++
 tests/run-lfs-symbols.sh | 27 +++++++++++++++------------
 3 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 44ea1c44..d0d32a87 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,9 @@
+2020-03-23  Mark Wielaard  <mark@klomp.org>
+
+	* getphdrnum.c: Include config.h.
+	* run-lfs-symbols.sh: Also check that file exists. Add more logs.
+	Remove ebl modules check.
+
 2020-03-22  Omar Sandoval  <osandov@fb.com>
 	    Mark Wielaard  <mark@klomp.org>
 
diff --git a/tests/getphdrnum.c b/tests/getphdrnum.c
index 4d4bb397..07f75db0 100644
--- a/tests/getphdrnum.c
+++ b/tests/getphdrnum.c
@@ -1,3 +1,5 @@
+#include "config.h"
+
 #include <fcntl.h>
 #include <libelf.h>
 #include <stdio.h>
diff --git a/tests/run-lfs-symbols.sh b/tests/run-lfs-symbols.sh
index f0894405..b65391b6 100755
--- a/tests/run-lfs-symbols.sh
+++ b/tests/run-lfs-symbols.sh
@@ -46,41 +46,44 @@ makeprint() {
 }
 
 testrun_lfs() {
-  bad=$(testrun ${abs_top_builddir}/src/nm -u "$1" | awk "$LFS")
-  if [ -n "$bad" ]; then
-    echo "$1 contains non-lfs symbols:" $bad
+  echo "checking $1"
+  if [ -e "$1" ]; then
+    bad=$(testrun ${abs_top_builddir}/src/nm -u "$1" | awk "$LFS")
+    if [ -n "$bad" ]; then
+      echo "$1 contains non-lfs symbols:" $bad
+      exit_status=1
+    fi
+  else
+    echo "$1 doesn't exist"
     exit_status=1
   fi
 }
 
-# First sanity-check that LFS detection works.
+echo First sanity-check that LFS detection works.
 exit_status=0
 testrun_lfs ./testfile-nolfs
 if [ $exit_status -eq 0 ]; then
   echo "Didn't detect any problem with testfile-nolfs!"
   exit 99
 fi
+echo
 
 exit_status=0
 
-# Check all normal build targets.
+echo Check all normal build targets.
 for dir in libelf libdw libasm libcpu src; do
   dir=${abs_top_builddir}/$dir
   for program in $(makeprint PROGRAMS $dir); do
     testrun_lfs $dir/$program
   done
 done
+echo
 
-# Check all libebl modules.
-dir=${abs_top_builddir}/backends
-for module in $(makeprint modules $dir); do
-  testrun_lfs $dir/libebl_$module.so
-done
-
-# Check all test programs.
+echo Check all test programs.
 dir=${abs_builddir}
 for program in $(makeprint check_PROGRAMS $dir); do
   testrun_lfs $dir/$program
 done
+echo
 
 exit $exit_status
-- 
2.18.2

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/457

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, 2020-03-04 at 12:09 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/457
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-s390x

Hohum. Our buildbot worker vm got upgraded to a z15. Which has zlib
deflate instructions. Which valgrind doesn't support, causing:

gnu compressed kmods are unstripped
==27438== 
==27438== Process terminating with default action of signal 4 (SIGILL): dumping core
==27438==  Illegal opcode at address 0x4ACCC80
==27438==    at 0x4ACCC80: ??? (in /usr/lib64/libz.so.1.2.11)
==27438==    by 0x4ABF05B: deflateResetKeep (in /usr/lib64/libz.so.1.2.11)
==27438==    by 0x4ABF0C5: deflateReset (in /usr/lib64/libz.so.1.2.11)
==27438==    by 0x4AC21FF: deflateInit_ (in /usr/lib64/libz.so.1.2.11)
==27438==    by 0x4856B35: __libelf_compress (elf_compress.c:113)
==27438==    by 0x485770B: elf_compress_gnu (elf_compress_gnu.c:104)
==27438==    by 0x100413D: compress_section (elfcompress.c:204)
==27438==    by 0x100318B: process_file (elfcompress.c:778)
==27438==    by 0x100318B: main (elfcompress.c:1359)

Luckily upstream valgrind already has a patch. But that will need to be
backported and then a new package should be installed on the s390x
buildbot worker.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/16/builds/277

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-aarch64

Build Reason: <unknown>
Blamelist: Konrad Kleine <kkleine@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Wed, Feb 26, 2020 at 09:42:26PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/16/builds/277
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-aarch64
> 
> Build Reason: <unknown>
> Blamelist: Konrad Kleine <kkleine@redhat.com>
> 
> BUILD FAILED: failed test (failure)

This is really strange. It doesn't seem to be related to the latest
commit. It seems to be some internal valgrind issue with the address
space manager:
https://builder.wildebeest.org/buildbot/#/builders/16/builds/277/steps/8/logs/test-suite_log

FAIL: run-nm-self.sh
====================
--18155:0: aspacem Valgrind: FATAL: aspacem assertion failed:
--18155:0: aspacem   sane_NSegment(&nsegments[i])
--18155:0: aspacem   at m_aspacemgr/aspacemgr-linux.c:728 (preen_nsegments)
--18155:0: aspacem Exiting now.

I'll try to replicate it and figure this out. But I think it doesn't
have to do anything with the elfutils code.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/269

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Mon, 2020-02-10 at 22:34 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/15/builds/269

This is a somewhat older failure and newer ones did show all green.
I didn't see the fix being posted to the list, so just for the record
here it is:

commit 990955ef0a320a73ed583246f23d5f220e5916f8
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Mon Feb 10 14:33:54 2020 -0500

    debuginfod testing: SIGUSR2 vs "groom" metric synch
    
    Previous code did not account for a groom job that was already
    completed at startup, so the SIGUSR2-triggered one may not have
    completed in time for the test.  The shell won the race condition on
    most buildbot VMs, but a debian builder showed the error of our ways.
    
    Signed-off-by: Frank Ch. Eigler <fche@redhat.com>

diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index 939826cd..1cc8f406 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -269,7 +269,8 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir, for a l
 rm -r R/debuginfod-rpms/rhel6/*
 kill -USR2 $PID1  # groom cycle
 # Expect 3 rpms to be deleted by the groom
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# 1 groom already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
 wait_ready $PORT1 'groom{statistic="file d/e"}' 3
 
 rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/441

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/417

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/415

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/462

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/457

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/461

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/458

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/429

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/254

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed compile (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/16/builds/251

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-aarch64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/412

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/425

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

Hi,

On Tue, 2019-12-03 at 09:49 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/4/builds/425
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-i386
> 
> Build Reason: <unknown>
> Blamelist: Mark Wielaard <mark@klomp.org>
> 
> BUILD FAILED: failed test (failure)

Note that this was a build failure some (8) days ago. The ppc64 builder
was offline and the buildbot only sends emails once all arches have
processed a build. The issue was simply a missing curl binary (which is
used in the run-debuginfod-find.sh testcase. It has since been
installed on the i386 worker. And everything is GREEN again.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/419

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/420

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/422

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/390

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Frank Ch. Eigler <fche@redhat.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/376

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, 2019-10-26 at 02:47 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/10/builds/376
> 

It looks like the s390x buildbot worker was just overloaded and timed
out during a compile job. The next build did succeed and everything
looks green.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/398

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/2/builds/393

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-amd64

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/397

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/4/builds/394

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-i386

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/363

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/350

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/348

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Ulrich Drepper <drepper@gmail.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Sat, Sep 07, 2019 at 02:54:33, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/398
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/

This happens during make distcheck.  It is two issues.

First the new test files weren't in EXTRA_DIST causing them to be not
available after make dist. This is fixed by the fist attached patch.

Second when building with the gcc undefined checker (configure
--enable-sanitize-undefined) the sanitizer finds one issue that is
undefined:

libcpu/riscv_disasm.c:457:57: runtime error: left shift of negative value -1

The issue is that the left shift is done on a signed value.  Simply
changing the context to be unsigned (as is done in the rest of the
code) fixes this. This is the second attached patch.

Note that the s390x builder was just taken offline for repairs. It
should come back online later this weekend. So it might take a bit for
the builder to be completely green again.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Add-new-riscv64-test-files-to-EXTRA_DIST.patch --]
[-- Type: text/x-diff, Size: 1411 bytes --]

From 1ee98b6c28fbe9ab532ef947cfe46e833535303c Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 7 Sep 2019 21:35:10 +0200
Subject: [PATCH] tests: Add new riscv64 test files to EXTRA_DIST

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog   | 5 +++++
 tests/Makefile.am | 4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 87c23102..b01ff965 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2019-09-07  Mark Wielaard  <mark@klomp.org>
+
+	* Makefile.am (EXTRA_DIST): Add run-disasm-riscv64.sh,
+	testfile-riscv64-dis1.o.bz2 and testfile-riscv64-dis1.expect.bz2.
+
 2019-08-27  Mark Wielaard  <mark@klomp.org>
 
 	* run-readelf-test2.sh: Add -x num testcase.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index d87d9616..ad0855de 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -438,7 +438,9 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
 	     run-readelf-discr.sh \
 	     testfile-rng.debug.bz2 testfile-urng.debug.bz2 \
 	     run-dwelf_elf_e_machine_string.sh \
-	     run-elfclassify.sh run-elfclassify-self.sh
+	     run-elfclassify.sh run-elfclassify-self.sh \
+	     run-disasm-riscv64.sh \
+	     testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2
 
 if USE_VALGRIND
 valgrind_cmd='valgrind -q --leak-check=full --error-exitcode=1'
-- 
2.20.1


[-- Attachment #3: 0001-libcpu-Make-sure-left-shifts-are-done-in-unsigned-co.patch --]
[-- Type: text/x-diff, Size: 1466 bytes --]

From 65a29626e327b0173174cb25f390426e7808d272 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 7 Sep 2019 21:45:26 +0200
Subject: [PATCH] libcpu: Make sure left shifts are done in unsigned context.

Use UINT64_C (0) instead of INT64_C (0) to make sure the calculation
is done on unsigned values. Otherwise the gcc undefined sanitizer will
warn:

libcpu/riscv_disasm.c:457:57: runtime error: left shift of negative value -1

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 libcpu/ChangeLog      | 5 +++++
 libcpu/riscv_disasm.c | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/libcpu/ChangeLog b/libcpu/ChangeLog
index 883896a2..e23097bd 100644
--- a/libcpu/ChangeLog
+++ b/libcpu/ChangeLog
@@ -1,3 +1,8 @@
+2019-09-07  Mark Wielaard  <mark@klomp.org>
+
+	* riscv_disasm.c (riscv_disasm): Use UINT64_C to make calculation
+	unsigned.
+
 2019-07-05  Omar Sandoval  <osandov@fb.com>
 
 	* Makefile.am: Combine libcpu_{i386,x86_64,bpf}.a into libcpu.a.
diff --git a/libcpu/riscv_disasm.c b/libcpu/riscv_disasm.c
index bc4e02e5..bc0d8f37 100644
--- a/libcpu/riscv_disasm.c
+++ b/libcpu/riscv_disasm.c
@@ -453,7 +453,7 @@ riscv_disasm (Ebl *ebl,
 	      mne = "fsd";
 	      break;
 	    case 16:
-	      opaddr = (((INT64_C (0) - ((first >> 12) & 0x1)) << 11)
+	      opaddr = (((UINT64_C (0) - ((first >> 12) & 0x1)) << 11)
 			| ((first << 2) & 0x400)
 			| ((first >> 1) & 0x300)
 			| ((first << 1) & 0x80)
-- 
2.20.1

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/16/builds/182

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-aarch64

Build Reason: <unknown>
Blamelist: Omar Sandoval <osandov@fb.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Thu, 2019-08-29 at 13:59 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/16/builds/182

Well that is unfortunate. The make check (under valgrind) just timed
out. That arm64 machine is really slow. And looking at some of the
other builders I see make check under valgrind (or make distcheck,
which enables the gcc undefined sanitizer and valgrind) became a lot
slower (non-valgrind builds/checks don't seem to be impacted).

It looks like the cause is our self checks. The majority of files we
use for that seem to be exactly those files that became a lot bigger.
Which means the tests use a lot more memory and increases the runtime
(which gets 10x longer under valgrind).

I am replacing some of the self test files with smaller
executables/libraries as attached. Hopefully that will reduce the make
check runtime under valgrind so builder don't time out.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Use-smaller-self-test-files.patch --]
[-- Type: text/x-patch, Size: 1586 bytes --]

From bb106065dc6962fabf1c163f18c932650dac6580 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Thu, 29 Aug 2019 16:21:35 +0200
Subject: [PATCH] tests: Use smaller self test files.

Don't use the largest executables/libraries to reduce the make check time.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog    | 6 ++++++
 tests/test-subr.sh | 8 ++++----
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 657e73b2..69e43ca8 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,9 @@
+2019-08-29  Mark Wielaard  <mark@klomp.org>
+
+	* test-subr.sh (self_test_files_exe): replace elfcmp, objdump and
+	readelf with elfclassify, stack and unstrip.
+	(self_test_files_lib): Replace libdw.so with libasm.so.
+
 2019-07-05  Omar Sandoval  <osandov@fb.com>
 
 	* Makefile.am: Remove -ldl.
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index e23a0176..e768c1e5 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -116,12 +116,12 @@ program_transform()
 }
 
 self_test_files_exe=`echo ${abs_top_builddir}/src/addr2line \
-${abs_top_builddir}/src/elfcmp \
-${abs_top_builddir}/src/objdump \
-${abs_top_builddir}/src/readelf`
+${abs_top_builddir}/src/elfclassify \
+${abs_top_builddir}/src/stack \
+${abs_top_builddir}/src/unstrip`
 
 self_test_files_lib=`echo ${abs_top_builddir}/libelf/libelf.so \
-${abs_top_builddir}/libdw/libdw.so`
+${abs_top_builddir}/libasm/libasm.so`
 
 self_test_files_obj=`echo ${abs_top_builddir}/src/size.o \
 ${abs_top_builddir}/src/strip.o`
-- 
2.18.1

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

On Thu, Aug 29, 2019 at 04:23:24PM +0200, Mark Wielaard wrote:
> I am replacing some of the self test files with smaller
> executables/libraries as attached. Hopefully that will reduce the make
> check runtime under valgrind so builder don't time out.

That was interesting. It did bring down the make check times under
valgrind. But it also showed a memory leak in eu-nm! Triggered by
chance because we use different self-check binaries now :)

Fix attached and pushed to master.

Cheers,

Mark

[-- Attachment #2: 0001-nm-Fix-latent-memory-leak-in-show_symbols.patch --]
[-- Type: text/x-diff, Size: 1786 bytes --]

From df33285b60290fadefd140ee2fe616f750105d2f Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Thu, 29 Aug 2019 17:46:52 +0200
Subject: [PATCH] nm: Fix latent memory leak in show_symbols.

If there are just a handful of symbols then memory for them is
allocated on the stack, otherwise the memory is malloced. So before
freeing the memory we need to check the number of entries to know if
the memory was heap allocated or not. But since not all entries might
be used we might have decreased the number of entries to the number
we will actually show. Remember the original symbol entries to not
have a memory leak.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 src/ChangeLog | 5 +++++
 src/nm.c      | 3 ++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/ChangeLog b/src/ChangeLog
index aeb62328..cb64f7d9 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,8 @@
+2019-08-26  Mark Wielaard  <mark@klomp.org>
+
+	* nm.c (show_symbols): Remember nentries_orig and check before
+	freeing sym_mem.
+
 2019-07-05  Omar Sandoval  <osandov@fb.com>
 
 	* Makefile.am: Remove -ldl.
diff --git a/src/nm.c b/src/nm.c
index da1350b4..7f6cf2a2 100644
--- a/src/nm.c
+++ b/src/nm.c
@@ -1438,6 +1438,7 @@ show_symbols (int fd, Ebl *ebl, GElf_Ehdr *ehdr,
   free (demangle_buffer);
 #endif
   /* Now we know the exact number.  */
+  size_t nentries_orig = nentries;
   nentries = nentries_used;
 
   /* Sort the entries according to the users wishes.  */
@@ -1472,7 +1473,7 @@ show_symbols (int fd, Ebl *ebl, GElf_Ehdr *ehdr,
     }
 
   /* Free all memory.  */
-  if (nentries * sizeof (sym_mem[0]) >= MAX_STACK_ALLOC)
+  if (nentries_orig * sizeof (sym_mem[0]) >= MAX_STACK_ALLOC)
     free (sym_mem);
 
   obstack_free (&whereob, NULL);
-- 
2.20.1

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/390

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Omar Sandoval <osandov@fb.com>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Wed, 2019-08-28 at 22:08 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/390
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/

That is an --enable-sanitize-undefined. I am not entirely clear on why
it only fails when linming with -fsanitize=undefined. But libasm.so
should also be linked with libebl_pic.a.

Which is what the attached fix does.

Cheers,

Mark

[-- Attachment #2: 0001-libasm-Link-against-libebl_pic.a.patch --]
[-- Type: text/x-patch, Size: 1311 bytes --]

From ab415cea199547ee06c50aa82eebe9c58307576f Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Wed, 28 Aug 2019 23:55:24 +0200
Subject: [PATCH] libasm: Link against libebl_pic.a.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 libasm/ChangeLog   | 4 ++++
 libasm/Makefile.am | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/libasm/ChangeLog b/libasm/ChangeLog
index 05b830c8..a1abac88 100644
--- a/libasm/ChangeLog
+++ b/libasm/ChangeLog
@@ -1,3 +1,7 @@
+2019-08-28  Mark Wielaard  <mark@klomp.org>
+
+	* Makefile.am (libasm_so_DEPS): Replace libebl.a with libebl_pic.a.
+
 2019-03-06  Mark Wielaard  <mark@klomp.org>
 
 	* asm_end.c (binary_end): Check return value of gelf_update_ehdr.
diff --git a/libasm/Makefile.am b/libasm/Makefile.am
index 19fef508..b2bff929 100644
--- a/libasm/Makefile.am
+++ b/libasm/Makefile.am
@@ -55,7 +55,7 @@ libasm_a_SOURCES = asm_begin.c asm_abort.c asm_end.c asm_error.c \
 libasm_pic_a_SOURCES =
 am_libasm_pic_a_OBJECTS = $(libasm_a_SOURCES:.c=.os)
 
-libasm_so_DEPS = ../lib/libeu.a ../libebl/libebl.a ../libelf/libelf.so ../libdw/libdw.so
+libasm_so_DEPS = ../lib/libeu.a ../libebl/libebl_pic.a ../libelf/libelf.so ../libdw/libdw.so
 libasm_so_LDLIBS = $(libasm_so_DEPS)
 if USE_LOCKS
 libasm_so_LDLIBS += -lpthread
-- 
2.18.1

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/10/builds/349

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-s390x

Build Reason: <unknown>
Blamelist: Dmitry V. Levin <ldv@altlinux.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/1/builds/376

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: centos-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Fri, Jun 28, 2019 at 11:02:02PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a failed build on builder whole buildset while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/1/builds/376
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: centos-x86_64

The test-suite.log says:

FAIL: run-large-elf-file.sh
===========================

long_bit: 64
space_available: 38
mem_available: 7
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.72662 s, 288 MB/s
addsections 2 testfile38 2147483648
add_sections 'testfile38': 2 (sec_size: 2147483648)
setshstrndx: 10
strip -o testfile38.strip -f testfile38.debug testfile38
elflint --gnu testfile38.strip
No errors
elflint --gnu -d testfile38.debug
No errors
unstrip testfile38.strip testfile38.debug
elfcmp  testfile38.strip
addsections 2 testfile27 2147483648
add_sections 'testfile27': 2 (sec_size: 2147483648)
setshstrndx: 30
strip -o testfile27.strip -f testfile27.debug testfile27
/srv/buildbot/worker/elfutils-centos-x86_64/build/elfutils-0.176/_build/src/strip: while writing 'testfile27.debug.SjHGU1': out of memory

And the config.log has:

=====================================================================
        elfutils: 0.176 (eu_version: 176)
=====================================================================

    Prefix                             : /srv/buildbot/worker/elfutils-centos-x8
6_64/build/elfutils-0.176/_inst
    Program prefix ("eu-" recommended) : eu-
    Source code location               : ..
    Maintainer mode                    : 
    libebl modules subdirectory        : elfutils
    build arch                         : x86_64-unknown-linux-gnu

  RECOMMENDED FEATURES (should all be yes)
    gzip support                       : yes
    bzip2 support                      : yes
    lzma/xz support                    : yes
    libstdc++ demangle support         : yes
    File textrel check                 : yes
    Symbol versioning                  : yes

  NOT RECOMMENDED FEATURES (should all be no)
    Experimental thread safety         : no
    install elf.h                      : no

  OTHER FEATURES
    Deterministic archives by default  : false
    Native language support            : yes

  EXTRA TEST FEATURES (used with make check)
    have bunzip2 installed (required)  : yes
    debug branch prediction            : no
    gprof support                      : no
    gcov support                       : no
    run all tests under valgrind       : yes
    gcc undefined behaviour sanitizer  : no
    use rpath in tests                 : no
    test biarch                        : yes

So, this is running the test under valgrind.
In which case we might need even more free memory available.

The test did succeed earlier on the worker, not under valgrind.

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

On Sat, 2019-06-29 at 01:47 +0200, Mark Wielaard wrote:
> FAIL: run-large-elf-file.sh
> ===========================
> 
> long_bit: 64
> space_available: 38
> mem_available: 7
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 3.72662 s, 288 MB/s
> addsections 2 testfile38 2147483648
> add_sections 'testfile38': 2 (sec_size: 2147483648)
> setshstrndx: 10
> strip -o testfile38.strip -f testfile38.debug testfile38
> elflint --gnu testfile38.strip
> No errors
> elflint --gnu -d testfile38.debug
> No errors
> unstrip testfile38.strip testfile38.debug
> elfcmp  testfile38.strip
> addsections 2 testfile27 2147483648
> add_sections 'testfile27': 2 (sec_size: 2147483648)
> setshstrndx: 30
> strip -o testfile27.strip -f testfile27.debug testfile27
> /srv/buildbot/worker/elfutils-centos-x86_64/build/elfutils-
> 0.176/_build/src/strip: while writing 'testfile27.debug.SjHGU1': out
> of memory
> [...]
> So, this is running the test under valgrind.
> In which case we might need even more free memory available.
> 
> The test did succeed earlier on the worker, not under valgrind.

So I pushed the attached commit to just increase the amount of free
memory needed when running under valgrind.

Cheers,

Mrk

[-- Attachment #2: 0001-tests-Add-2GB-to-mem_needed-running-run-large-elf-fi.patch --]
[-- Type: text/x-patch, Size: 1661 bytes --]

From 31c8b3f098b0654db8f573b2a15d5b6d07d4d3b0 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 1 Jul 2019 17:08:43 +0200
Subject: [PATCH] tests: Add 2GB to mem_needed running run-large-elf-file.sh
 with valgrind

Running under valgrind just requires even more free memory.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog             |  5 +++++
 tests/run-large-elf-file.sh | 10 ++++++++--
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 9d15f8f..400588f 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2019-07-01  Mark Wielaard  <mark@klomp.org>
+
+	* run-large-elf-file.sh: Add 2GB to mem_needed when running under
+	valgrind.
+
 2019-06-18  Mark Wielaard  <mark@klomp.org>
 
 	* Makefile.am (TESTS): Add run-large-elf-file.sh.
diff --git a/tests/run-large-elf-file.sh b/tests/run-large-elf-file.sh
index c244c60..6146cfe 100755
--- a/tests/run-large-elf-file.sh
+++ b/tests/run-large-elf-file.sh
@@ -36,10 +36,16 @@ if test $space_available -lt 10; then
 fi
 
 # Make sure the files fit into memory, assume 6GB needed (2.5 * 2 + 1 extra).
+# Running under valgrind might need even more.
+mem_needed=6
+if [ "x$VALGRIND_CMD" != "x" ]; then
+  mem_needed=$[${mem_needed} + 2]
+fi
+echo "mem_needed: $mem_needed"
 mem_available=$(free -g | grep ^Mem: | awk -F ' +' '{print $7}')
 echo "mem_available: $mem_available"
-if test $mem_available -lt 6; then
-  echo "Need at least 6GB free available memory"
+if test $mem_available -lt $mem_needed; then
+  echo "Need at least ${mem_needed}GB free available memory"
   exit 77
 fi
 
-- 
1.8.3.1

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/299

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Tue, 2018-11-13 at 16:42 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/299

This is disappointing. You fix one bug and find another :)

The issue here is (when running the testsuite under gcc undefinedness
sanitizer):

runtest testfile-debug-rel-ppc64-z.o
../../../libelf/gelf_xlate.h:53:1: runtime error: member access within
misaligned address 0x00000402928d for type 'struct Elf64_Chdr', which
requires 8 byte alignment
0x00000402928d: note: pointer points here
 1d bb 0a c3 01 00 00  00 00 00 00 00 71 00 00  00 00 00 00 00 01 00
00  00 00 00 00 00 78 9c 63  60
             ^ 
*** failure strip testfile-debug-rel-ppc64-z.o
../../../libelf/gelf_xlate.h:53:1: runtime error: member access within
misaligned address 0x000004029175 for type 'struct Elf64_Chdr', which
requires 8 byte alignment
0x000004029175: note: pointer points here
 1d bb 0a c3 01 00 00  00 00 00 00 00 71 00 00  00 00 00 00 00 01 00
00  00 00 00 00 00 78 9c 63  60
             ^ 
*** failure strip --reloc-debug-sections testfile-debug-rel-ppc64-z.o

This happens when writing out the file to disk. The section is
compressed, and we try to write it back to disk, for this we need to
translate the on disk Elf_Chdr (because it is a big endian ELF file and
the memory representation on x86_64 is little endian):

#1  0x00007ffff7f8ad98 in Elf64_cvt_Chdr (encode=<optimized out>, 
    len=<optimized out>, src=<optimized out>, dest=<optimized out>)
    at gelf_xlate.h:37
#2  Elf64_cvt_chdr (dest=<optimized out>, src=0x439380, len=69, 
    encode=<optimized out>) at chdr_xlate.h:32
#3  0x00007ffff7f9fd96 in __elf64_updatemmap (elf=elf@entry=0x4377c0, 
    change_bo=change_bo@entry=1, shnum=shnum@entry=22)
    at elf32_updatefile.c:368
#4  0x00007ffff7f98a3d in write_file (shnum=22, change_bo=1, size=2872, 
    elf=0x4377c0) at elf_update.c:121
#5  elf_update (elf=0x4377c0, cmd=cmd@entry=ELF_C_WRITE) at elf_update.c:231
#6  0x000000000040bd29 in handle_elf (fd=3, elf=<optimized out>, mode=436, 
    tvp=0x0, prefix=0x0) at strip.c:2448
#7  0x000000000040f4be in process_file (
    fname=0x7fffffffdbab "testfile-debug-rel-ppc64-z.o") at strip.c:769
#8  0x0000000000402bdf in main (argc=5, argv=0x7fffffffd848) at strip.c:265

The compressed sections however simply have an alignment of 1.

I think this is really a bug in binutils gas which produced this
particular testfile. But it looks like we make the same mistake
(decompressing and recompressing with eu-elfcompress produces the same
misaligned compressed section).

So we might actually have two bugs. 1) we don't handle incorrectly
aligned compressed sections and 2) we generate such incorrectly aligned
compressed sections ourselves.

Pondering fixes.

Cheers,

Mark

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On Tue, 2018-11-13 at 18:29 +0100, Mark Wielaard wrote:
> So we might actually have two bugs. 1) we don't handle incorrectly
> aligned compressed sections and 2) we generate such incorrectly
> aligned compressed sections ourselves.
> 
> Pondering fixes.

The fixes were easy!
(Hopefully the buildbot agrees.)
I pushed the attached.

Cheers,

Mark

[-- Attachment #2: 0001-libelf-Correctly-setup-alignment-of-SHF_COMPRESSED-s.patch --]
[-- Type: text/x-patch, Size: 2801 bytes --]

From cf10453f8252df81225796d98548ba6eac113df3 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Tue, 13 Nov 2018 21:18:09 +0100
Subject: [PATCH] libelf: Correctly setup alignment of SHF_COMPRESSED section
 data.

We didn't set the alignment of SHF_COMPRESSED sections correctly.
Those sections start with an Elf(32|64)_Chdr. Make sure sh_addralign
is setup to be able to read such a struct directly. Likewise don't
trust the alignment set on any SHF_COMPRESSED section, but always
make the (raw) compressed data aligned correctly for the reading the
Elf(32|64)_Chdr directly.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 libelf/ChangeLog      |  7 +++++++
 libelf/elf_compress.c |  4 ++--
 libelf/elf_getdata.c  | 10 ++++++++--
 3 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/libelf/ChangeLog b/libelf/ChangeLog
index ab078cb..93820d1 100644
--- a/libelf/ChangeLog
+++ b/libelf/ChangeLog
@@ -1,3 +1,10 @@
+2018-11-13  Mark Wielaard  <mark@klomp.org>
+
+	* elf_getdata.c (__libelf_set_rawdata_wrlock): Explicitly set the
+	alignment of SHF_COMPRESSED data to the alignment of ELF_T_CHDR.
+	* elf_compress.c (elf_compress): After compression set sh_addralign
+	to the alignment of ELF_T_CHDR.
+
 2018-11-09  Mark Wielaard  <mark@klomp.org>
 
 	* elf_compress_gnu.c (elf_compress_gnu): Use elf_getdata.
diff --git a/libelf/elf_compress.c b/libelf/elf_compress.c
index d96245d..be9eeab 100644
--- a/libelf/elf_compress.c
+++ b/libelf/elf_compress.c
@@ -455,14 +455,14 @@ elf_compress (Elf_Scn *scn, int type, unsigned int flags)
 	{
 	  Elf32_Shdr *shdr = elf32_getshdr (scn);
 	  shdr->sh_size = new_size;
-	  shdr->sh_addralign = 1;
+	  shdr->sh_addralign = __libelf_type_align (ELFCLASS32, ELF_T_CHDR);
 	  shdr->sh_flags |= SHF_COMPRESSED;
 	}
       else
 	{
 	  Elf64_Shdr *shdr = elf64_getshdr (scn);
 	  shdr->sh_size = new_size;
-	  shdr->sh_addralign = 1;
+	  shdr->sh_addralign = __libelf_type_align (ELFCLASS64, ELF_T_CHDR);
 	  shdr->sh_flags |= SHF_COMPRESSED;
 	}
 
diff --git a/libelf/elf_getdata.c b/libelf/elf_getdata.c
index 2043bba..639a798 100644
--- a/libelf/elf_getdata.c
+++ b/libelf/elf_getdata.c
@@ -268,9 +268,15 @@ __libelf_set_rawdata_wrlock (Elf_Scn *scn)
       /* First a test whether the section is valid at all.  */
       size_t entsize;
 
-      /* Compressed data has a header, but then compressed data.  */
+      /* Compressed data has a header, but then compressed data.
+	 Make sure to set the alignment of the header explicitly,
+	 don't trust the file alignment for the section, it is
+	 often wrong.  */
       if ((flags & SHF_COMPRESSED) != 0)
-	entsize = 1;
+	{
+	  entsize = 1;
+	  align = __libelf_type_align (elf->class, ELF_T_CHDR);
+	}
       else if (type == SHT_HASH)
 	{
 	  GElf_Ehdr ehdr_mem;
-- 
1.8.3.1

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/3/builds/296

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-x86_64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot

Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard]

On Sat, 2018-11-10 at 12:31 +0000, buildbot@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/3/builds/296
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: fedora-x86_64

The buildbot fedora-x86_64 worker was just upgraded to Fedora 29.
This introduced '.gnu.build.attributes' ELF notes into the build files,
which make various self-tests barf.

I'll look into supporting these ELF notes.

Cheers,

Mark

Buildbot failure in Wildebeest Builder on whole buildset

[buildbot]

The Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/11/builds/244

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64le

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/12/builds/242

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: fedora-ppc64

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The BuildbotThe Buildbot has detected a failed build on builder whole buildset while building elfutils.
Full details are available at:
    https://builder.wildebeest.org/buildbot/#builders/15/builds/81

Buildbot URL: https://builder.wildebeest.org/buildbot/

Worker for this Build: debian-armhf

Build Reason: <unknown>
Blamelist: Mark Wielaard <mark@klomp.org>

BUILD FAILED: failed test (failure)

Sincerely,
 -The Buildbot