public inbox for gdb@sourceware.org
 help / color / mirror / Atom feed
* [builder] gdb_check_step: remove gdb.gdb/selftest.exp
@ 2022-06-08 18:51 Mark Wielaard
  2022-06-09  9:37 ` Luis Machado
  0 siblings, 1 reply; 13+ messages in thread
From: Mark Wielaard @ 2022-06-08 18:51 UTC (permalink / raw)
  To: buildbot; +Cc: gdb, Mark Wielaard

Hi,

I just committed the following update to builder:

    selftest.exp is bad (flaky), just keep the gdb and gdbserver
    unittest.exp for the CI runs.

builder.sourceware.org does two builds for gdb.  quick CI builds that
just build gdb and gdb-server and run a small amount of tests that are
known to reliably PASS. If any of this fails it will sent email to the
gdb-testers mailinglist and the author of the commit.
https://builder.sourceware.org/buildbot/#/builders?tags=gdb

It also does full builds for binutils-gdb as a whole plus the full
testsuite both against gcc and clang. These do record all logs in
bunsendb but doesn't sent emails:
https://builder.sourceware.org/buildbot/#/builders?tags=binutils-gdb

With the removal of the selftest.exp there are only two unittest.exp
tests (one for gdb and one for gdbserver). This does seem very
minimal. Could someone suggest stable, known good passing tests that
should be added because when they do start failing it really is a
regression that should be flagged?

Note that the intention is to also allow a trybot that runs the same
set of CI tests so it can be used as pre-commit check.

Cheers,

Mark

P.S. There were same bad failing emails today. Our apologies, that was
a bunsen upload typo.

---
 builder/master.cfg | 1 -
 1 file changed, 1 deletion(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index 852322f..af4f65f 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1446,7 +1446,6 @@ gdb_check_step = steps.Test(
                  util.Interpolate('-j%(prop:ncpus)s'),
                  'check-gdb',
                  ("TESTS= "
-                  "gdb.gdb/selftest.exp "
                   "gdb.gdb/unittest.exp "
                   "gdb.server/unittest.exp ")],
         name='make check-gdb',
-- 
2.30.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-08 18:51 [builder] gdb_check_step: remove gdb.gdb/selftest.exp Mark Wielaard
@ 2022-06-09  9:37 ` Luis Machado
  2022-06-09 23:09   ` Mark Wielaard
  0 siblings, 1 reply; 13+ messages in thread
From: Luis Machado @ 2022-06-09  9:37 UTC (permalink / raw)
  To: Mark Wielaard, buildbot; +Cc: gdb

On 6/8/22 19:51, Mark Wielaard wrote:
> Hi,
> 
> I just committed the following update to builder:
> 
>      selftest.exp is bad (flaky), just keep the gdb and gdbserver
>      unittest.exp for the CI runs.
> 
> builder.sourceware.org does two builds for gdb.  quick CI builds that
> just build gdb and gdb-server and run a small amount of tests that are
> known to reliably PASS. If any of this fails it will sent email to the
> gdb-testers mailinglist and the author of the commit.
> https://builder.sourceware.org/buildbot/#/builders?tags=gdb
> 
> It also does full builds for binutils-gdb as a whole plus the full
> testsuite both against gcc and clang. These do record all logs in
> bunsendb but doesn't sent emails:
> https://builder.sourceware.org/buildbot/#/builders?tags=binutils-gdb
> 
> With the removal of the selftest.exp there are only two unittest.exp
> tests (one for gdb and one for gdbserver). This does seem very
> minimal. Could someone suggest stable, known good passing tests that
> should be added because when they do start failing it really is a
> regression that should be flagged?
> 
> Note that the intention is to also allow a trybot that runs the same
> set of CI tests so it can be used as pre-commit check.
> 
> Cheers,
> 
> Mark
> 
> P.S. There were same bad failing emails today. Our apologies, that was
> a bunsen upload typo.
> 
> ---
>   builder/master.cfg | 1 -
>   1 file changed, 1 deletion(-)
> 
> diff --git a/builder/master.cfg b/builder/master.cfg
> index 852322f..af4f65f 100644
> --- a/builder/master.cfg
> +++ b/builder/master.cfg
> @@ -1446,7 +1446,6 @@ gdb_check_step = steps.Test(
>                    util.Interpolate('-j%(prop:ncpus)s'),
>                    'check-gdb',
>                    ("TESTS= "
> -                  "gdb.gdb/selftest.exp "
>                     "gdb.gdb/unittest.exp "
>                     "gdb.server/unittest.exp ")],
>           name='make check-gdb',

I always use gdb.base/break.exp as a good smoke test. If that one fails, then things
are really broken.

I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude
gdb.base/break-interp.exp, which is problematic on some targets.

If you also want to exercise gdbserver, then run with native-gdbserver and native-extended-gdbserver
boards. For example:

make check-gdb TESTS="gdb.base/break*.exp" RUNTESTFLAGS="--target_board=native-gdbserver" -j$(nproc).

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-09  9:37 ` Luis Machado
@ 2022-06-09 23:09   ` Mark Wielaard
  2022-06-09 23:21     ` Mark Wielaard
  0 siblings, 1 reply; 13+ messages in thread
From: Mark Wielaard @ 2022-06-09 23:09 UTC (permalink / raw)
  To: Luis Machado; +Cc: buildbot, gdb

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

Hi Luis,

On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > With the removal of the selftest.exp there are only two unittest.exp
> > tests (one for gdb and one for gdbserver). This does seem very
> > minimal. Could someone suggest stable, known good passing tests that
> > should be added because when they do start failing it really is a
> > regression that should be flagged?
> > 
> > Note that the intention is to also allow a trybot that runs the same
> > set of CI tests so it can be used as pre-commit check.
> [...] 
> I always use gdb.base/break.exp as a good smoke test. If that one fails, then things
> are really broken.
> 
> I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude
> gdb.base/break-interp.exp, which is problematic on some targets.

It never is just easy is it? :) You are right, I saw break-interp.exp
fail...  I tried to come up with a regexp but gave up given that it
has to go throug python first and then we don't know whether the
worker uses bash as /bin/sh so I just added them all (exclusing
break-interp.exp) as a list.

> If you also want to exercise gdbserver, then run with native-gdbserver and native-extended-gdbserver
> boards. For example:
> 
> make check-gdb TESTS="gdb.base/break*.exp" RUNTESTFLAGS="--target_board=native-gdbserver" -j$(nproc).

OK, so I added two more steps, one with
RUNTESTFLAGS="--target_board=native-gdbserver" and one with
RUNTESTFLAGS="--target_board=native-extended-gdbserver"

So now each CI run does make check-gdb three times.  Frank, you might
want to look at the bunsen upload code, I didn't know how to adapt it,
so it only records the logs of the first make check-gdb step.

Thanks,

Mark

[-- Attachment #2: 0001-gdb-Add-break-exp-TESTS-and-gdb_check_native_gdbserv.patch --]
[-- Type: text/x-diff, Size: 3899 bytes --]

From 9ef7c1e94e9a4f30acc5d8b320d4f7b44ff4cfef Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Fri, 10 Jun 2022 00:51:59 +0200
Subject: [PATCH] gdb: Add break*exp TESTS and
 gdb_check_native_gdbserver[_extended]_step

Add gdb.base/break*exp, but not break-interp.exp, as gdb_test_exp
list.  Add two more make gdb-check steps with RUNTESTFLAGS set to
--target_board=native-gdbserver and
--target_board=native-extended-gdbserver.
---
 builder/master.cfg | 48 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 45 insertions(+), 3 deletions(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index 647acd0..f6895b5 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1447,19 +1447,59 @@ gdb_make_step = steps.Compile(
         name='make',
         haltOnFailure=True)
 # Only a small subset of tests that are fast and known to PASS.
+gdb_test_exp = ("TESTS= "
+                "gdb.base/break-always.exp "
+                "gdb.base/break-caller-line.exp "
+                "gdb.base/break-entry.exp "
+                "gdb.base/break.exp "
+                "gdb.base/break-fun-addr.exp "
+                "gdb.base/break-idempotent.exp "
+                "gdb.base/break-include.exp "
+                "gdb.base/break-inline.exp "
+                "gdb.base/break-main-file-remove-fail.exp "
+                "gdb.base/break-on-linker-gcd-function.exp "
+                "gdb.base/breakpoint-in-ro-region.exp "
+                "gdb.base/breakpoint-shadow.exp "
+                "gdb.base/break-probes.exp "
+                "gdb.base/break-unload-file.exp "
+                "gdb.gdb/unittest.exp "
+                "gdb.server/unittest.exp ")
+
 gdb_check_step = steps.Test(
         workdir='gdb-build',
         command=['make',
                  util.Interpolate('-j%(prop:ncpus)s'),
                  'check-gdb',
-                 ("TESTS= "
-                  "gdb.gdb/unittest.exp "
-                  "gdb.server/unittest.exp ")],
+                 gdb_test_exp],
         name='make check-gdb',
         logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum",
                    "gdb.log": "gdb/testsuite/gdb.log" },
         haltOnFailure=False, flunkOnFailure=True)
 
+gdb_check_native_gdbserver_step = steps.Test(
+        workdir='gdb-build',
+        command=['make',
+                 util.Interpolate('-j%(prop:ncpus)s'),
+                 'check-gdb',
+                 gdb_test_exp,
+                 'RUNTESTFLAGS="--target_board=native-gdbserver"'],
+        name='make check-gdb native-gdbserver',
+        logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum",
+                   "gdb.log": "gdb/testsuite/gdb.log" },
+        haltOnFailure=False, flunkOnFailure=True)
+
+gdb_check_native_extended_gdbserver_step = steps.Test(
+        workdir='gdb-build',
+        command=['make',
+                 util.Interpolate('-j%(prop:ncpus)s'),
+                 'check-gdb',
+                 gdb_test_exp,
+                 'RUNTESTFLAGS="--target_board=native-extended-gdbserver"'],
+        name='make check-gdb native-extended-gdbserver',
+        logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum",
+                   "gdb.log": "gdb/testsuite/gdb.log" },
+        haltOnFailure=False, flunkOnFailure=True)
+
 gdb_factory = util.BuildFactory()
 gdb_factory.addStep(gdb_git_step)
 gdb_factory.addStep(gdb_rm_step)
@@ -1468,6 +1508,8 @@ gdb_factory.addStep(gdb_make_step)
 gdb_factory.addStep(gdb_check_step)
 gdb_factory.addSteps(bunsen_logfile_upload_steps([ # only a few tests being run but still
         (["../gdb-build/gdb/testsuite/gdb.sum", "../gdb-build/gdb/testsuite/gdb.log", "../gdb-build/gdb/config.log"],"gdb")]))
+gdb_factory.addStep(gdb_check_native_gdbserver_step)
+gdb_factory.addStep(gdb_check_native_extended_gdbserver_step)
 
 # Used for armhf only because of
 # https://sourceware.org/bugzilla/show_bug.cgi?id=28561
-- 
2.30.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-09 23:09   ` Mark Wielaard
@ 2022-06-09 23:21     ` Mark Wielaard
  2022-06-10 10:50       ` Mark Wielaard
  0 siblings, 1 reply; 13+ messages in thread
From: Mark Wielaard @ 2022-06-09 23:21 UTC (permalink / raw)
  To: Luis Machado; +Cc: gdb, buildbot

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
> On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > I always use gdb.base/break.exp as a good smoke test. If that one fails, then things
> > are really broken.
> > 
> > I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude
> > gdb.base/break-interp.exp, which is problematic on some targets.
> 
> It never is just easy is it? :) You are right, I saw break-interp.exp
> fail...  I tried to come up with a regexp but gave up given that it
> has to go throug python first and then we don't know whether the
> worker uses bash as /bin/sh so I just added them all (exclusing
> break-interp.exp) as a list.

Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes fails.
I have removed from the list. Hopefully the remaining list does actually pass.

Cheers,

Mark



[-- Attachment #2: 0001-gdb_test_exp-Remove-gdb.base-break-unload-file.exp.patch --]
[-- Type: text/x-diff, Size: 764 bytes --]

From 49a95a50543a772d4413078c34de520e2e2a9610 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Fri, 10 Jun 2022 01:20:19 +0200
Subject: [PATCH] gdb_test_exp: Remove gdb.base/break-unload-file.exp

---
 builder/master.cfg | 1 -
 1 file changed, 1 deletion(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index f6895b5..e413d3f 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1461,7 +1461,6 @@ gdb_test_exp = ("TESTS= "
                 "gdb.base/breakpoint-in-ro-region.exp "
                 "gdb.base/breakpoint-shadow.exp "
                 "gdb.base/break-probes.exp "
-                "gdb.base/break-unload-file.exp "
                 "gdb.gdb/unittest.exp "
                 "gdb.server/unittest.exp ")
 
-- 
2.30.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-09 23:21     ` Mark Wielaard
@ 2022-06-10 10:50       ` Mark Wielaard
  2022-06-10 10:58         ` Luis Machado
  0 siblings, 1 reply; 13+ messages in thread
From: Mark Wielaard @ 2022-06-10 10:50 UTC (permalink / raw)
  To: Luis Machado; +Cc: gdb, buildbot

Hi,

On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
> > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > > I always use gdb.base/break.exp as a good smoke test. If that one
> > > fails, then things
> > > are really broken.
> > > 
> > > I think gdb.base/break*.exp should make a good smoke test list.
> > > We just need to exclude
> > > gdb.base/break-interp.exp, which is problematic on some targets.
> > 
> > It never is just easy is it? :) You are right, I saw break-
> > interp.exp
> > fail...  I tried to come up with a regexp but gave up given that it
> > has to go throug python first and then we don't know whether the
> > worker uses bash as /bin/sh so I just added them all (exclusing
> > break-interp.exp) as a list.
> 
> Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes
> fails.
> I have removed from the list. Hopefully the remaining list does
> actually pass.

And it didn't :{

Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) 
gdb.base/break-idempotent.exp under both native-gdbserver and native-
extended-gdbserver
https://builder.sourceware.org/buildbot/#builders/76/builds/446
https://builder.sourceware.org/buildbot/#builders/85/builds/294

So I have removed that one too from the list.
The CI test list now looks like:

# Only a small subset of tests that are fast and known to PASS.
gdb_test_exp = ("TESTS= "
                "gdb.base/break-always.exp "
                "gdb.base/break-caller-line.exp "
                "gdb.base/break-entry.exp "
                "gdb.base/break.exp "
                "gdb.base/break-fun-addr.exp "
                "gdb.base/break-include.exp "
                "gdb.base/break-inline.exp "
                "gdb.base/break-main-file-remove-fail.exp "
                "gdb.base/break-on-linker-gcd-function.exp "
                "gdb.base/breakpoint-in-ro-region.exp "
                "gdb.base/breakpoint-shadow.exp "
                "gdb.base/break-probes.exp "
                "gdb.gdb/unittest.exp "
                "gdb.server/unittest.exp ")

Which will be run three times with make gdb-check, once without a
target_board, once with native-gdbserver and once with native-extended-
gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-arm64,
fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64,
opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-check
because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561)
I'll add a debian-i386 builder so there is more 32bit coverage.

All are green now (with the latest change to remove break-idempotent)
https://builder.sourceware.org/buildbot/#/builders?tags=gdb

Question is if this is a good list, does it need more tests? And should
it maybe be maintained in the binutils-gdb repo instead of in the
builder repo?

For example we could have a make check-gdb-ci target which does what
the buildbot would do (and then the buildbot could just call that).

Cheers,

Mark

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 10:50       ` Mark Wielaard
@ 2022-06-10 10:58         ` Luis Machado
  2022-06-10 15:17           ` will schmidt
  2022-06-10 15:54           ` Carl Love
  0 siblings, 2 replies; 13+ messages in thread
From: Luis Machado @ 2022-06-10 10:58 UTC (permalink / raw)
  To: Mark Wielaard, Carl Love, Will Schmidt; +Cc: gdb, buildbot

On 6/10/22 11:50, Mark Wielaard wrote:
> Hi,
> 
> On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
>> On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
>>> On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
>>>> I always use gdb.base/break.exp as a good smoke test. If that one
>>>> fails, then things
>>>> are really broken.
>>>>
>>>> I think gdb.base/break*.exp should make a good smoke test list.
>>>> We just need to exclude
>>>> gdb.base/break-interp.exp, which is problematic on some targets.
>>>
>>> It never is just easy is it? :) You are right, I saw break-
>>> interp.exp
>>> fail...  I tried to come up with a regexp but gave up given that it
>>> has to go throug python first and then we don't know whether the
>>> worker uses bash as /bin/sh so I just added them all (exclusing
>>> break-interp.exp) as a list.
>>
>> Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes
>> fails.
>> I have removed from the list. Hopefully the remaining list does
>> actually pass.
> 
> And it didn't :{
> 

Yeah. As expected, the GDB testsuite is a bit delicate when you start dealing with
multiple architectures and modes. But I think this is good progress already.

> Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED)
> gdb.base/break-idempotent.exp under both native-gdbserver and native-
> extended-gdbserver
> https://builder.sourceware.org/buildbot/#builders/76/builds/446
> https://builder.sourceware.org/buildbot/#builders/85/builds/294

Those might be genuine issues. I'm cc-ing Carl and Will so they can chime in.

> 
> So I have removed that one too from the list.
> The CI test list now looks like:
> 
> # Only a small subset of tests that are fast and known to PASS.
> gdb_test_exp = ("TESTS= "
>                  "gdb.base/break-always.exp "
>                  "gdb.base/break-caller-line.exp "
>                  "gdb.base/break-entry.exp "
>                  "gdb.base/break.exp "
>                  "gdb.base/break-fun-addr.exp "
>                  "gdb.base/break-include.exp "
>                  "gdb.base/break-inline.exp "
>                  "gdb.base/break-main-file-remove-fail.exp "
>                  "gdb.base/break-on-linker-gcd-function.exp "
>                  "gdb.base/breakpoint-in-ro-region.exp "
>                  "gdb.base/breakpoint-shadow.exp "
>                  "gdb.base/break-probes.exp "
>                  "gdb.gdb/unittest.exp "
>                  "gdb.server/unittest.exp ")
> 
> Which will be run three times with make gdb-check, once without a
> target_board, once with native-gdbserver and once with native-extended-
> gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-arm64,
> fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64,
> opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-check
> because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561)
> I'll add a debian-i386 builder so there is more 32bit coverage.
> 
> All are green now (with the latest change to remove break-idempotent)
> https://builder.sourceware.org/buildbot/#/builders?tags=gdb

Looks nice! :-)

> 
> Question is if this is a good list, does it need more tests? And should
> it maybe be maintained in the binutils-gdb repo instead of in the
> builder repo?
> 
> For example we could have a make check-gdb-ci target which does what
> the buildbot would do (and then the buildbot could just call that).

Having a new check-gdb target that only does minimal smoke tests should
be easy to do. Once we determine a subset of critical tests, we can put something
together if folks think it is a good idea. I like it, as it make it easier to deal with
stability issues of GDB's testsuite across different targets.

> 
> Cheers,
> 
> Mark


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 10:58         ` Luis Machado
@ 2022-06-10 15:17           ` will schmidt
  2022-06-10 15:54           ` Carl Love
  1 sibling, 0 replies; 13+ messages in thread
From: will schmidt @ 2022-06-10 15:17 UTC (permalink / raw)
  To: Luis Machado, Mark Wielaard, Carl Love; +Cc: gdb, buildbot

On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote:
> On 6/10/22 11:50, Mark Wielaard wrote:
> > Hi,
> > 
> > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
> > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > > > > I always use gdb.base/break.exp as a good smoke test. If that
> > > > > one
> > > > > fails, then things
> > > > > are really broken.
> > > > > 
> > > > > I think gdb.base/break*.exp should make a good smoke test
> > > > > list.
> > > > > We just need to exclude
> > > > > gdb.base/break-interp.exp, which is problematic on some
> > > > > targets.
> > > > 
> > > > It never is just easy is it? :) You are right, I saw break-
> > > > interp.exp
> > > > fail...  I tried to come up with a regexp but gave up given
> > > > that it
> > > > has to go throug python first and then we don't know whether
> > > > the
> > > > worker uses bash as /bin/sh so I just added them all (exclusing
> > > > break-interp.exp) as a list.
> > > 
> > > Sigh, sorry, looks like gdb.base/break-unload-file.exp also
> > > sometimes
> > > fails.
> > > I have removed from the list. Hopefully the remaining list does
> > > actually pass.
> > 
> > And it didn't :{
> > 
> 
> Yeah. As expected, the GDB testsuite is a bit delicate when you start
> dealing with
> multiple architectures and modes. But I think this is good progress
> already.
> 
> > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED)
> > gdb.base/break-idempotent.exp under both native-gdbserver and
> > native-
> > extended-gdbserver
> > https://builder.sourceware.org/buildbot/#builders/76/builds/446
> > https://builder.sourceware.org/buildbot/#builders/85/builds/294
> 
> Those might be genuine issues. I'm cc-ing Carl and Will so they can
> chime in.


RROR: no fileid for debian-ppc64-builder
UNRESOLVED: gdb.base/break-idempotent.exp: pie=nopie:
always_inserted=off: watch: gdb_breakpoint: set breakpoint at main
(timeout)

I've looked briefly at the logs.  I recall one of those tests having a
FAIL with one of the test permutations, but not an UNRESOLVED.  We'll
need to poke further.

has_hw_wp_support: Default, hardware watchpoint
not deteced

I see a typo in there somewhere... :-)   





> 
> > So I have removed that one too from the list.
> > The CI test list now looks like:
> > 
> > # Only a small subset of tests that are fast and known to PASS.
> > gdb_test_exp = ("TESTS= "
> >                  "gdb.base/break-always.exp "
> >                  "gdb.base/break-caller-line.exp "
> >                  "gdb.base/break-entry.exp "
> >                  "gdb.base/break.exp "
> >                  "gdb.base/break-fun-addr.exp "
> >                  "gdb.base/break-include.exp "
> >                  "gdb.base/break-inline.exp "
> >                  "gdb.base/break-main-file-remove-fail.exp "
> >                  "gdb.base/break-on-linker-gcd-function.exp "
> >                  "gdb.base/breakpoint-in-ro-region.exp "
> >                  "gdb.base/breakpoint-shadow.exp "
> >                  "gdb.base/break-probes.exp "
> >                  "gdb.gdb/unittest.exp "
> >                  "gdb.server/unittest.exp ")
> > 
> > Which will be run three times with make gdb-check, once without a
> > target_board, once with native-gdbserver and once with native-
> > extended-
> > gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-
> > arm64,
> > fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64,
> > opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-
> > check
> > because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561)
> > I'll add a debian-i386 builder so there is more 32bit coverage.
> > 
> > All are green now (with the latest change to remove break-
> > idempotent)
> > https://builder.sourceware.org/buildbot/#/builders?tags=gdb
> 
> Looks nice! :-)
> 
> > Question is if this is a good list, does it need more tests? And
> > should
> > it maybe be maintained in the binutils-gdb repo instead of in the
> > builder repo?
> > 
> > For example we could have a make check-gdb-ci target which does
> > what
> > the buildbot would do (and then the buildbot could just call that).
> 
> Having a new check-gdb target that only does minimal smoke tests
> should
> be easy to do. Once we determine a subset of critical tests, we can
> put something
> together if folks think it is a good idea. I like it, as it make it
> easier to deal with
> stability issues of GDB's testsuite across different targets.
> 
> > Cheers,
> > 
> > Mark


^ permalink raw reply	[flat|nested] 13+ messages in thread

* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 10:58         ` Luis Machado
  2022-06-10 15:17           ` will schmidt
@ 2022-06-10 15:54           ` Carl Love
  2022-06-10 19:11             ` Mark Wielaard
  1 sibling, 1 reply; 13+ messages in thread
From: Carl Love @ 2022-06-10 15:54 UTC (permalink / raw)
  To: Luis Machado, Mark Wielaard, Will Schmidt; +Cc: gdb, buildbot

On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote:
> On 6/10/22 11:50, Mark Wielaard wrote:
> > Hi,
> > 
> > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
> > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > > > > I always use gdb.base/break.exp as a good smoke test. If that
> > > > > one
> > > > > fails, then things
> > > > > are really broken.
> > > > > 
> > > > > I think gdb.base/break*.exp should make a good smoke test
> > > > > list.
> > > > > We just need to exclude
> > > > > gdb.base/break-interp.exp, which is problematic on some
> > > > > targets.

Trying to to understand how you are running the test.  I think you are
running these as remote tests, i.e. machine A requests that a remote
machine B run the test via gdbserver, correct?
 
I tried running the tests naitively, i.e. on a Power 10  RHEL 9 system
I ran:

make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break-
interp.exp  '

The gdb summary is:

                === gdb Summary ===

# of expected passes            702
# of expected failures          20
> > > > 
> > > > It never is just easy is it? :) You are right, I saw break-
> > > > interp.exp
> > > > fail...  I tried to come up with a regexp but gave up given
> > > > that it
> > > > has to go throug python first and then we don't know whether
> > > > the
> > > > worker uses bash as /bin/sh so I just added them all (exclusing
> > > > break-interp.exp) as a list.
> > > 
> > > Sigh, sorry, looks like gdb.base/break-unload-file.exp also
> > > sometimes
> > > fails.
> > > I have removed from the list. Hopefully the remaining list does
> > > actually pass.

I ran gdb.base/break-unload-file.exp on the same Power 10 RHEL 9
system. 

make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break-unload-
file.exp  '

 The gdb summary is:
                === gdb Summary ===

# of expected passes            36

So it doesn't appear the tests are fundamentally broken.

We have only played around with running the gdb testsuite remotely a
little.  Specifically, we were working on documenting how it is done. 
We see a number of additional issues running remotely versus natively
but at this point have not had the time/bandwidth to dig into them

> > 
> > And it didn't :{
> > 
> 
> Yeah. As expected, the GDB testsuite is a bit delicate when you start
> dealing with
> multiple architectures and modes. But I think this is good progress
> already.
> 
> > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED)
> > gdb.base/break-idempotent.exp under both native-gdbserver and
> > native-
> > extended-gdbserver

Running that test natively, i.e.

 make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break-
idempotent.exp  ' 

generates a number of errors:
 
ERROR: breakpoints not deleted
ERROR: no fileid for ltcd97-lp2
ERROR: Couldn't send delete breakpoints to GDB.
ERROR: can't read "gdb_spawn_id": no such variable
    while executing
"expect {
-i 1000 -timeout 100 
	-re ".*A problem internal to GDB has been detected" {
	    fail "$message (GDB internal error)"
	    gdb_internal_erro..."
    ("uplevel" body line 1)
    invoked from within
"uplevel $body" TCL READ VARNAME can't read "gdb_spa

The gdb summary is:

                === gdb Summary ===

# of expected passes            35
# of unresolved testcases       3



> > https://builder.sourceware.org/buildbot/#builders/76/builds/446 
> > https://builder.sourceware.org/buildbot/#builders/85/builds/294 
> 
> Those might be genuine issues. I'm cc-ing Carl and Will so they can
> chime in.
> 
Need to work on getting this one fixed for a native run before worrying
about running it remotely.

I will add this to the list of gdb issues that needs work.

                           Carl


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 15:54           ` Carl Love
@ 2022-06-10 19:11             ` Mark Wielaard
  2022-06-10 20:11               ` Carl Love
  2022-06-29 15:58               ` Carl Love
  0 siblings, 2 replies; 13+ messages in thread
From: Mark Wielaard @ 2022-06-10 19:11 UTC (permalink / raw)
  To: Carl Love; +Cc: Luis Machado, Will Schmidt, gdb, buildbot

Hi Carl,

On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote:
> On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote:
> > On 6/10/22 11:50, Mark Wielaard wrote:
> > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote:
> > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote:
> > > > > > I always use gdb.base/break.exp as a good smoke test. If that
> > > > > > one
> > > > > > fails, then things
> > > > > > are really broken.
> > > > > > 
> > > > > > I think gdb.base/break*.exp should make a good smoke test
> > > > > > list.
> > > > > > We just need to exclude
> > > > > > gdb.base/break-interp.exp, which is problematic on some
> > > > > > targets.
> 
> Trying to to understand how you are running the test.  I think you are
> running these as remote tests, i.e. machine A requests that a remote
> machine B run the test via gdbserver, correct?

Yes, but machine A and B are the same one in this case.  The tests are
run three times. Once with just gdb, and then with a target-board set
to either native-gdbserver or native-extended-gdbserver.

The buildbot log stdio should show you the exact settings.

https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio

make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp gdb.base/break-probes.exp gdb.gdb/unittest.exp gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native-gdbserver"'

So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that tells
the test to run against the native-gdbserver.

Cheers,

Mark


^ permalink raw reply	[flat|nested] 13+ messages in thread

* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 19:11             ` Mark Wielaard
@ 2022-06-10 20:11               ` Carl Love
  2022-06-10 22:23                 ` Frank Ch. Eigler
  2022-06-29 15:58               ` Carl Love
  1 sibling, 1 reply; 13+ messages in thread
From: Carl Love @ 2022-06-10 20:11 UTC (permalink / raw)
  To: Mark Wielaard; +Cc: Luis Machado, Will Schmidt, gdb, buildbot

Mark:

On Fri, 2022-06-10 at 21:11 +0200, Mark Wielaard wrote:
> Hi Carl,
> 
> On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote:
> > On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote:
> > > On 6/10/22 11:50, Mark Wielaard wrote:
> > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> > > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard
> > > > > wrote:
> > > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado
> > > > > > wrote:
> > > > > > > I always use gdb.base/break.exp as a good smoke test. If
> > > > > > > that
> > > > > > > one
> > > > > > > fails, then things
> > > > > > > are really broken.
> > > > > > > 
> > > > > > > I think gdb.base/break*.exp should make a good smoke test
> > > > > > > list.
> > > > > > > We just need to exclude
> > > > > > > gdb.base/break-interp.exp, which is problematic on some
> > > > > > > targets.
> > 
> > Trying to to understand how you are running the test.  I think you
> > are
> > running these as remote tests, i.e. machine A requests that a
> > remote
> > machine B run the test via gdbserver, correct?
> 
> Yes, but machine A and B are the same one in this case.  The tests
> are
> run three times. Once with just gdb, and then with a target-board set
> to either native-gdbserver or native-extended-gdbserver.
> 
> The buildbot log stdio should show you the exact settings.
> 
> https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio 
> 
> make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-
> caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp
> gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp
> gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-
> main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp
> gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp
> gdb.base/break-probes.exp gdb.gdb/unittest.exp
> gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native-
> gdbserver"'
> 
> So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that
> tells
> the test to run against the native-gdbserver.

OK, thanks that will be helpful for testing. 

So, I have been trying to isolate where the messages 
  
   ERROR: no fileid for ltcd97-lp2
   ERROR: Couldn't send delete breakpoints to GDB.
   ERROR: can't read "gdb_spawn_id": no such variable

Looks like a couple of the error messages are coming from proc
default_gdb_exit.  The routine is called as part of the check in the
original test program.  The break-idempotent.exp calls
skip_hw_watchpoint_tests  which calls has_hw_wp_support  which calls
gdb_exit which calls proc default_gdb_exit.  In default_gdb_exit we
have:

   if ![is_remote host] {

        remote_close host     <-   generates ERROR: no fileid for ltcd97-lp2
    }
    unset gdb_spawn_id        <-   generates ERROR: can't read "gdb_spawn_id": no such variable
    unset ::gdb_tty_name

    unset inferior_spawn_id

So, now to figure out why those statements are failing....???

                   Carl




^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 20:11               ` Carl Love
@ 2022-06-10 22:23                 ` Frank Ch. Eigler
  0 siblings, 0 replies; 13+ messages in thread
From: Frank Ch. Eigler @ 2022-06-10 22:23 UTC (permalink / raw)
  To: Carl Love; +Cc: Mark Wielaard, gdb, buildbot, Will Schmidt, Luis Machado

Hi -

As an aside: now that these triplicated test results are starting to
flow properly into bunsen, you can try having it do some comparisons
for you:

    https://tinyurl.com/bdz4xyv8

to show the list of recent test/build runs from this batch.  Can click
on any subset's checkbox, hit "diff", and from there go to the
respective runs' logfile extracts.

- FChE

^ permalink raw reply	[flat|nested] 13+ messages in thread

* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-10 19:11             ` Mark Wielaard
  2022-06-10 20:11               ` Carl Love
@ 2022-06-29 15:58               ` Carl Love
  2022-06-29 22:42                 ` Mark Wielaard
  1 sibling, 1 reply; 13+ messages in thread
From: Carl Love @ 2022-06-29 15:58 UTC (permalink / raw)
  To: Mark Wielaard, cel; +Cc: Luis Machado, Will Schmidt, gdb, buildbot

Mark:

The test was failing the same way when running locally.  It was a test
issue related to checking for hardware watchpoint support on PowerPC. I
posted a patch to fix it.  Andrew Burgess merged a similar patch that
he did to fix the test with my patch and committed the fix.  The test
is now running correctly locally.  Note, I have not tested it in the
remote configuration as you mentioned.   Here is the commit for the
fix:

commit 13f72372413400410aaa94b7f0e2ff7de663fdcb
Author: Carl Love <cel@us.ibm.com>
Date:   Wed Jun 22 16:14:20 2022 +0100

    gdb/testsuite: fix gdb.base/break-idempotent.exp on ppc
    
    When running the gdb.base/break-idempotent.exp test on ppc, I was
    seeing some test failures (or rather errors), that looked like
this:
    
      (gdb) watch local
      Hardware watchpoint 2: local
    
      has_hw_wp_support: Hardware watchpoint detected
      ERROR: no fileid for gcc2-power8
      ERROR: Couldn't send delete breakpoints to GDB.
      ERROR OCCURED: can't read "gdb_spawn_id": no such variable
          while executing
...

Please let me know if you see any additional issues with the test. 
Thanks.

                    Carl 


On Fri, 2022-06-10 at 21:11 +0200, Mark Wielaard wrote:
> Hi Carl,
> 
> On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote:
> > On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote:
> > > On 6/10/22 11:50, Mark Wielaard wrote:
> > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote:
> > > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard
> > > > > wrote:
> > > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado
> > > > > > wrote:
> > > > > > > I always use gdb.base/break.exp as a good smoke test. If
> > > > > > > that
> > > > > > > one
> > > > > > > fails, then things
> > > > > > > are really broken.
> > > > > > > 
> > > > > > > I think gdb.base/break*.exp should make a good smoke test
> > > > > > > list.
> > > > > > > We just need to exclude
> > > > > > > gdb.base/break-interp.exp, which is problematic on some
> > > > > > > targets.
> > 
> > Trying to to understand how you are running the test.  I think you
> > are
> > running these as remote tests, i.e. machine A requests that a
> > remote
> > machine B run the test via gdbserver, correct?
> 
> Yes, but machine A and B are the same one in this case.  The tests
> are
> run three times. Once with just gdb, and then with a target-board set
> to either native-gdbserver or native-extended-gdbserver.
> 
> The buildbot log stdio should show you the exact settings.
> 
> https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio 
> 
> make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-
> caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp
> gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp
> gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-
> main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp
> gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp
> gdb.base/break-probes.exp gdb.gdb/unittest.exp
> gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native-
> gdbserver"'
> 
> So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that
> tells
> the test to run against the native-gdbserver.
> 
> Cheers,
> 
> Mark
> 


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp
  2022-06-29 15:58               ` Carl Love
@ 2022-06-29 22:42                 ` Mark Wielaard
  0 siblings, 0 replies; 13+ messages in thread
From: Mark Wielaard @ 2022-06-29 22:42 UTC (permalink / raw)
  To: Carl Love; +Cc: Luis Machado, Will Schmidt, gdb, buildbot

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

Hi Carl,

On Wed, Jun 29, 2022 at 08:58:43AM -0700, Carl Love wrote:
> The test was failing the same way when running locally.  It was a test
> issue related to checking for hardware watchpoint support on PowerPC. I
> posted a patch to fix it.  Andrew Burgess merged a similar patch that
> he did to fix the test with my patch and committed the fix.  The test
> is now running correctly locally.  Note, I have not tested it in the
> remote configuration as you mentioned.   Here is the commit for the
> fix:
> 
> commit 13f72372413400410aaa94b7f0e2ff7de663fdcb
> Author: Carl Love <cel@us.ibm.com>
> Date:   Wed Jun 22 16:14:20 2022 +0100
> 
>     gdb/testsuite: fix gdb.base/break-idempotent.exp on ppc
>     
>     When running the gdb.base/break-idempotent.exp test on ppc, I was
>     seeing some test failures (or rather errors), that looked like
> this:
>     
>       (gdb) watch local
>       Hardware watchpoint 2: local
>     
>       has_hw_wp_support: Hardware watchpoint detected
>       ERROR: no fileid for gcc2-power8
>       ERROR: Couldn't send delete breakpoints to GDB.
>       ERROR OCCURED: can't read "gdb_spawn_id": no such variable
>           while executing
> ...
> 
> Please let me know if you see any additional issues with the test. 

Lets add it back to the buildbot tests and see if anything breaks.

Cheers,

Mark

[-- Attachment #2: 0001-gdb_test_exp-Add-gdb.base-break-idempotent.exp-back.patch --]
[-- Type: text/x-diff, Size: 844 bytes --]

From a1f7675ab2fa8a4c987d0e7f1fb53de564cf9ed9 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Thu, 30 Jun 2022 00:40:08 +0200
Subject: [PATCH] gdb_test_exp: Add gdb.base/break-idempotent.exp back

This should now work on all arches
---
 builder/master.cfg | 1 +
 1 file changed, 1 insertion(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index 219f88d..729ec37 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1820,6 +1820,7 @@ gdb_test_exp = ("TESTS= "
                 "gdb.base/break-entry.exp "
                 "gdb.base/break.exp "
                 "gdb.base/break-fun-addr.exp "
+                "gdb.base/break-idempotent.exp "
                 "gdb.base/break-include.exp "
                 "gdb.base/break-inline.exp "
                 "gdb.base/break-main-file-remove-fail.exp "
-- 
2.30.2


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2022-06-29 22:42 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-08 18:51 [builder] gdb_check_step: remove gdb.gdb/selftest.exp Mark Wielaard
2022-06-09  9:37 ` Luis Machado
2022-06-09 23:09   ` Mark Wielaard
2022-06-09 23:21     ` Mark Wielaard
2022-06-10 10:50       ` Mark Wielaard
2022-06-10 10:58         ` Luis Machado
2022-06-10 15:17           ` will schmidt
2022-06-10 15:54           ` Carl Love
2022-06-10 19:11             ` Mark Wielaard
2022-06-10 20:11               ` Carl Love
2022-06-10 22:23                 ` Frank Ch. Eigler
2022-06-29 15:58               ` Carl Love
2022-06-29 22:42                 ` Mark Wielaard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).