[Bug external/31575] New: [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[Bug external/31575] New: [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

            Bug ID: 31575
           Summary: [gdb/external, linaro CI] progressions and regressions
                    reported on unrelated commits
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: external
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

I submitted a patch series with 3 test-case fixes here (
https://sourceware.org/pipermail/gdb-patches/2024-March/207598.html ).

Then I got the following message from the linaro CI:
...
Dear contributor, our automatic CI has detected problems related to your
patch(es).  Please find some details below.  If you have any questions, please
follow up on linaro-toolchain@lists.linaro.org mailing list, Libera's
#linaro-tcwg channel, or ping your favourite Linaro toolchain developer on the
usual project channel.

We appreciate that it might be difficult to find the necessary logs or
reproduce the issue locally. If you can't get what you need from our CI within
minutes, let us know and we will be happy to help.

In gdb_check master-arm after:

  | 2 patches in gdb
  | Patchwork URL: https://patchwork.sourceware.org/patch/87702
  | 0ba222c3fdd Fix missing return type in gdb.linespec/break-asm-file.c
  | fd739928eeb Add missing include in gdb.base/ctf-ptype.c
  | ... applied on top of baseline commit:
  | b58829cdeff x86/SSE2AVX: move checking

FAIL: 8 regressions: 16 progressions

regressions.sum:
                === gdb tests ===

Running gdb:gdb.ada/verylong.exp ...
FAIL: gdb.ada/verylong.exp: print (x / 4) * 2
FAIL: gdb.ada/verylong.exp: print +x
FAIL: gdb.ada/verylong.exp: print -x
FAIL: gdb.ada/verylong.exp: print x
FAIL: gdb.ada/verylong.exp: print x - 99 + 1
FAIL: gdb.ada/verylong.exp: print x / 2
FAIL: gdb.ada/verylong.exp: print x = 170141183460469231731687303715884105727
... and 4 more entries

progressions.sum:
                === gdb tests ===

Running gdb:gdb.ada/convvar_comp.exp ...
FAIL: gdb.ada/convvar_comp.exp: print $item.started
FAIL: gdb.ada/convvar_comp.exp: set variable $item := item
FAIL: gdb.ada/convvar_comp.exp: print item.started

Running gdb:gdb.ada/enum_idx_packed.exp ...
FAIL: gdb.ada/enum_idx_packed.exp: scenario=minimal: print multi'first
FAIL: gdb.ada/enum_idx_packed.exp: scenario=minimal: print multi_multi
... and 18 more entries

You can find the failure logs in *.log.1.xz files in
 -
https://ci.linaro.org/job/tcwg_gdb_check--master-arm-precommit/2021/artifact/artifacts/artifacts.precommit/00-sumfiles/
The full lists of regressions and progressions as well as configure and make
commands are in
 -
https://ci.linaro.org/job/tcwg_gdb_check--master-arm-precommit/2021/artifact/artifacts/artifacts.precommit/notify/
The list of [ignored] baseline and flaky failures are in
 -
https://ci.linaro.org/job/tcwg_gdb_check--master-arm-precommit/2021/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail

The configuration of this build is:
CI config tcwg_gdb_check master-arm

-----------------8<--------------------------8<--------------------------8<--------------------------
The information below can be used to reproduce a debug environment:

Current build   :
https://ci.linaro.org/job/tcwg_gdb_check--master-arm-precommit/2021/artifact/artifacts
Reference build :
https://ci.linaro.org/job/tcwg_gdb_check--master-arm-build/983/artifact/artifacts

Warning: we do not enable maintainer-mode nor automatically update
generated files, which may lead to failures if the patch modifies the
master files.
...

Clearly the test-cases for which regressions and progressions are claimed have
nothing to do with the submitted patches, which touch two unrelated .c files in
the test suite.

I managed to reproduce the FAILs in gdb.ada/verylong.exp, and filed PR31574.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug external/31575] [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maxim.kuvyrkov at linaro dot org,
                   |                            |thiago.bauermann at linaro dot org

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
Is there something that can be done about these false positives?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug external/31575] [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[maxim.kuvyrkov at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

Maxim Kuvyrkov <maxim.kuvyrkov at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maxim.kuvyrkov at gmail dot com

--- Comment #2 from Maxim Kuvyrkov <maxim.kuvyrkov at gmail dot com> ---
Hi Tom,

Yes, we have noticed the problem this morning.  Working on it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug external/31575] [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Maxim Kuvyrkov from comment #2)
> Hi Tom,
> 
> Yes, we have noticed the problem this morning.  Working on it.

Thanks :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug external/31575] [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[maxim.kuvyrkov at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

--- Comment #4 from Maxim Kuvyrkov <maxim.kuvyrkov at gmail dot com> ---
This was caused by a transition side-effect.

We have recently fixed install of system gnat compiler, which enabled more GDB
tests to run.  Unfortunately, the baseline run didn't have the fix, so baseline
results didn't have the additional tests (and additional FAILs!):
UNSUPPORTED: gdb.ada/verylong.exp: require failed: gnatmake_version_at_least 11

The pre-commit builds picked up the fix, enabled additional testsuites, and
reported additional FAILs as regressions.

We'll monitor our CI today, and investigate further if we see more failures.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug external/31575] [gdb/external, linaro CI] progressions and regressions reported on unrelated commits

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31575

--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Maxim Kuvyrkov from comment #4)
> This was caused by a transition side-effect.
> 
> We have recently fixed install of system gnat compiler, which enabled more
> GDB tests to run.  Unfortunately, the baseline run didn't have the fix, so
> baseline results didn't have the additional tests (and additional FAILs!):
> UNSUPPORTED: gdb.ada/verylong.exp: require failed: gnatmake_version_at_least
> 11
> 
> The pre-commit builds picked up the fix, enabled additional testsuites, and
> reported additional FAILs as regressions.
> 
> We'll monitor our CI today, and investigate further if we see more failures.

Sounds like a mechanism is required to ensure that only comparable results are
compared.

-- 
You are receiving this mail because:
You are on the CC list for the bug.