* [builder] gdb_check_step: remove gdb.gdb/selftest.exp @ 2022-06-08 18:51 Mark Wielaard 2022-06-09 9:37 ` Luis Machado 0 siblings, 1 reply; 13+ messages in thread From: Mark Wielaard @ 2022-06-08 18:51 UTC (permalink / raw) To: buildbot; +Cc: gdb, Mark Wielaard Hi, I just committed the following update to builder: selftest.exp is bad (flaky), just keep the gdb and gdbserver unittest.exp for the CI runs. builder.sourceware.org does two builds for gdb. quick CI builds that just build gdb and gdb-server and run a small amount of tests that are known to reliably PASS. If any of this fails it will sent email to the gdb-testers mailinglist and the author of the commit. https://builder.sourceware.org/buildbot/#/builders?tags=gdb It also does full builds for binutils-gdb as a whole plus the full testsuite both against gcc and clang. These do record all logs in bunsendb but doesn't sent emails: https://builder.sourceware.org/buildbot/#/builders?tags=binutils-gdb With the removal of the selftest.exp there are only two unittest.exp tests (one for gdb and one for gdbserver). This does seem very minimal. Could someone suggest stable, known good passing tests that should be added because when they do start failing it really is a regression that should be flagged? Note that the intention is to also allow a trybot that runs the same set of CI tests so it can be used as pre-commit check. Cheers, Mark P.S. There were same bad failing emails today. Our apologies, that was a bunsen upload typo. --- builder/master.cfg | 1 - 1 file changed, 1 deletion(-) diff --git a/builder/master.cfg b/builder/master.cfg index 852322f..af4f65f 100644 --- a/builder/master.cfg +++ b/builder/master.cfg @@ -1446,7 +1446,6 @@ gdb_check_step = steps.Test( util.Interpolate('-j%(prop:ncpus)s'), 'check-gdb', ("TESTS= " - "gdb.gdb/selftest.exp " "gdb.gdb/unittest.exp " "gdb.server/unittest.exp ")], name='make check-gdb', -- 2.30.2 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-08 18:51 [builder] gdb_check_step: remove gdb.gdb/selftest.exp Mark Wielaard @ 2022-06-09 9:37 ` Luis Machado 2022-06-09 23:09 ` Mark Wielaard 0 siblings, 1 reply; 13+ messages in thread From: Luis Machado @ 2022-06-09 9:37 UTC (permalink / raw) To: Mark Wielaard, buildbot; +Cc: gdb On 6/8/22 19:51, Mark Wielaard wrote: > Hi, > > I just committed the following update to builder: > > selftest.exp is bad (flaky), just keep the gdb and gdbserver > unittest.exp for the CI runs. > > builder.sourceware.org does two builds for gdb. quick CI builds that > just build gdb and gdb-server and run a small amount of tests that are > known to reliably PASS. If any of this fails it will sent email to the > gdb-testers mailinglist and the author of the commit. > https://builder.sourceware.org/buildbot/#/builders?tags=gdb > > It also does full builds for binutils-gdb as a whole plus the full > testsuite both against gcc and clang. These do record all logs in > bunsendb but doesn't sent emails: > https://builder.sourceware.org/buildbot/#/builders?tags=binutils-gdb > > With the removal of the selftest.exp there are only two unittest.exp > tests (one for gdb and one for gdbserver). This does seem very > minimal. Could someone suggest stable, known good passing tests that > should be added because when they do start failing it really is a > regression that should be flagged? > > Note that the intention is to also allow a trybot that runs the same > set of CI tests so it can be used as pre-commit check. > > Cheers, > > Mark > > P.S. There were same bad failing emails today. Our apologies, that was > a bunsen upload typo. > > --- > builder/master.cfg | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/builder/master.cfg b/builder/master.cfg > index 852322f..af4f65f 100644 > --- a/builder/master.cfg > +++ b/builder/master.cfg > @@ -1446,7 +1446,6 @@ gdb_check_step = steps.Test( > util.Interpolate('-j%(prop:ncpus)s'), > 'check-gdb', > ("TESTS= " > - "gdb.gdb/selftest.exp " > "gdb.gdb/unittest.exp " > "gdb.server/unittest.exp ")], > name='make check-gdb', I always use gdb.base/break.exp as a good smoke test. If that one fails, then things are really broken. I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude gdb.base/break-interp.exp, which is problematic on some targets. If you also want to exercise gdbserver, then run with native-gdbserver and native-extended-gdbserver boards. For example: make check-gdb TESTS="gdb.base/break*.exp" RUNTESTFLAGS="--target_board=native-gdbserver" -j$(nproc). ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-09 9:37 ` Luis Machado @ 2022-06-09 23:09 ` Mark Wielaard 2022-06-09 23:21 ` Mark Wielaard 0 siblings, 1 reply; 13+ messages in thread From: Mark Wielaard @ 2022-06-09 23:09 UTC (permalink / raw) To: Luis Machado; +Cc: buildbot, gdb [-- Attachment #1: Type: text/plain, Size: 1714 bytes --] Hi Luis, On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > With the removal of the selftest.exp there are only two unittest.exp > > tests (one for gdb and one for gdbserver). This does seem very > > minimal. Could someone suggest stable, known good passing tests that > > should be added because when they do start failing it really is a > > regression that should be flagged? > > > > Note that the intention is to also allow a trybot that runs the same > > set of CI tests so it can be used as pre-commit check. > [...] > I always use gdb.base/break.exp as a good smoke test. If that one fails, then things > are really broken. > > I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude > gdb.base/break-interp.exp, which is problematic on some targets. It never is just easy is it? :) You are right, I saw break-interp.exp fail... I tried to come up with a regexp but gave up given that it has to go throug python first and then we don't know whether the worker uses bash as /bin/sh so I just added them all (exclusing break-interp.exp) as a list. > If you also want to exercise gdbserver, then run with native-gdbserver and native-extended-gdbserver > boards. For example: > > make check-gdb TESTS="gdb.base/break*.exp" RUNTESTFLAGS="--target_board=native-gdbserver" -j$(nproc). OK, so I added two more steps, one with RUNTESTFLAGS="--target_board=native-gdbserver" and one with RUNTESTFLAGS="--target_board=native-extended-gdbserver" So now each CI run does make check-gdb three times. Frank, you might want to look at the bunsen upload code, I didn't know how to adapt it, so it only records the logs of the first make check-gdb step. Thanks, Mark [-- Attachment #2: 0001-gdb-Add-break-exp-TESTS-and-gdb_check_native_gdbserv.patch --] [-- Type: text/x-diff, Size: 3899 bytes --] From 9ef7c1e94e9a4f30acc5d8b320d4f7b44ff4cfef Mon Sep 17 00:00:00 2001 From: Mark Wielaard <mark@klomp.org> Date: Fri, 10 Jun 2022 00:51:59 +0200 Subject: [PATCH] gdb: Add break*exp TESTS and gdb_check_native_gdbserver[_extended]_step Add gdb.base/break*exp, but not break-interp.exp, as gdb_test_exp list. Add two more make gdb-check steps with RUNTESTFLAGS set to --target_board=native-gdbserver and --target_board=native-extended-gdbserver. --- builder/master.cfg | 48 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/builder/master.cfg b/builder/master.cfg index 647acd0..f6895b5 100644 --- a/builder/master.cfg +++ b/builder/master.cfg @@ -1447,19 +1447,59 @@ gdb_make_step = steps.Compile( name='make', haltOnFailure=True) # Only a small subset of tests that are fast and known to PASS. +gdb_test_exp = ("TESTS= " + "gdb.base/break-always.exp " + "gdb.base/break-caller-line.exp " + "gdb.base/break-entry.exp " + "gdb.base/break.exp " + "gdb.base/break-fun-addr.exp " + "gdb.base/break-idempotent.exp " + "gdb.base/break-include.exp " + "gdb.base/break-inline.exp " + "gdb.base/break-main-file-remove-fail.exp " + "gdb.base/break-on-linker-gcd-function.exp " + "gdb.base/breakpoint-in-ro-region.exp " + "gdb.base/breakpoint-shadow.exp " + "gdb.base/break-probes.exp " + "gdb.base/break-unload-file.exp " + "gdb.gdb/unittest.exp " + "gdb.server/unittest.exp ") + gdb_check_step = steps.Test( workdir='gdb-build', command=['make', util.Interpolate('-j%(prop:ncpus)s'), 'check-gdb', - ("TESTS= " - "gdb.gdb/unittest.exp " - "gdb.server/unittest.exp ")], + gdb_test_exp], name='make check-gdb', logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum", "gdb.log": "gdb/testsuite/gdb.log" }, haltOnFailure=False, flunkOnFailure=True) +gdb_check_native_gdbserver_step = steps.Test( + workdir='gdb-build', + command=['make', + util.Interpolate('-j%(prop:ncpus)s'), + 'check-gdb', + gdb_test_exp, + 'RUNTESTFLAGS="--target_board=native-gdbserver"'], + name='make check-gdb native-gdbserver', + logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum", + "gdb.log": "gdb/testsuite/gdb.log" }, + haltOnFailure=False, flunkOnFailure=True) + +gdb_check_native_extended_gdbserver_step = steps.Test( + workdir='gdb-build', + command=['make', + util.Interpolate('-j%(prop:ncpus)s'), + 'check-gdb', + gdb_test_exp, + 'RUNTESTFLAGS="--target_board=native-extended-gdbserver"'], + name='make check-gdb native-extended-gdbserver', + logfiles={ "gdb.sum": "gdb/testsuite/gdb.sum", + "gdb.log": "gdb/testsuite/gdb.log" }, + haltOnFailure=False, flunkOnFailure=True) + gdb_factory = util.BuildFactory() gdb_factory.addStep(gdb_git_step) gdb_factory.addStep(gdb_rm_step) @@ -1468,6 +1508,8 @@ gdb_factory.addStep(gdb_make_step) gdb_factory.addStep(gdb_check_step) gdb_factory.addSteps(bunsen_logfile_upload_steps([ # only a few tests being run but still (["../gdb-build/gdb/testsuite/gdb.sum", "../gdb-build/gdb/testsuite/gdb.log", "../gdb-build/gdb/config.log"],"gdb")])) +gdb_factory.addStep(gdb_check_native_gdbserver_step) +gdb_factory.addStep(gdb_check_native_extended_gdbserver_step) # Used for armhf only because of # https://sourceware.org/bugzilla/show_bug.cgi?id=28561 -- 2.30.2 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-09 23:09 ` Mark Wielaard @ 2022-06-09 23:21 ` Mark Wielaard 2022-06-10 10:50 ` Mark Wielaard 0 siblings, 1 reply; 13+ messages in thread From: Mark Wielaard @ 2022-06-09 23:21 UTC (permalink / raw) To: Luis Machado; +Cc: gdb, buildbot [-- Attachment #1: Type: text/plain, Size: 888 bytes --] On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > I always use gdb.base/break.exp as a good smoke test. If that one fails, then things > > are really broken. > > > > I think gdb.base/break*.exp should make a good smoke test list. We just need to exclude > > gdb.base/break-interp.exp, which is problematic on some targets. > > It never is just easy is it? :) You are right, I saw break-interp.exp > fail... I tried to come up with a regexp but gave up given that it > has to go throug python first and then we don't know whether the > worker uses bash as /bin/sh so I just added them all (exclusing > break-interp.exp) as a list. Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes fails. I have removed from the list. Hopefully the remaining list does actually pass. Cheers, Mark [-- Attachment #2: 0001-gdb_test_exp-Remove-gdb.base-break-unload-file.exp.patch --] [-- Type: text/x-diff, Size: 764 bytes --] From 49a95a50543a772d4413078c34de520e2e2a9610 Mon Sep 17 00:00:00 2001 From: Mark Wielaard <mark@klomp.org> Date: Fri, 10 Jun 2022 01:20:19 +0200 Subject: [PATCH] gdb_test_exp: Remove gdb.base/break-unload-file.exp --- builder/master.cfg | 1 - 1 file changed, 1 deletion(-) diff --git a/builder/master.cfg b/builder/master.cfg index f6895b5..e413d3f 100644 --- a/builder/master.cfg +++ b/builder/master.cfg @@ -1461,7 +1461,6 @@ gdb_test_exp = ("TESTS= " "gdb.base/breakpoint-in-ro-region.exp " "gdb.base/breakpoint-shadow.exp " "gdb.base/break-probes.exp " - "gdb.base/break-unload-file.exp " "gdb.gdb/unittest.exp " "gdb.server/unittest.exp ") -- 2.30.2 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-09 23:21 ` Mark Wielaard @ 2022-06-10 10:50 ` Mark Wielaard 2022-06-10 10:58 ` Luis Machado 0 siblings, 1 reply; 13+ messages in thread From: Mark Wielaard @ 2022-06-10 10:50 UTC (permalink / raw) To: Luis Machado; +Cc: gdb, buildbot Hi, On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > > I always use gdb.base/break.exp as a good smoke test. If that one > > > fails, then things > > > are really broken. > > > > > > I think gdb.base/break*.exp should make a good smoke test list. > > > We just need to exclude > > > gdb.base/break-interp.exp, which is problematic on some targets. > > > > It never is just easy is it? :) You are right, I saw break- > > interp.exp > > fail... I tried to come up with a regexp but gave up given that it > > has to go throug python first and then we don't know whether the > > worker uses bash as /bin/sh so I just added them all (exclusing > > break-interp.exp) as a list. > > Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes > fails. > I have removed from the list. Hopefully the remaining list does > actually pass. And it didn't :{ Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) gdb.base/break-idempotent.exp under both native-gdbserver and native- extended-gdbserver https://builder.sourceware.org/buildbot/#builders/76/builds/446 https://builder.sourceware.org/buildbot/#builders/85/builds/294 So I have removed that one too from the list. The CI test list now looks like: # Only a small subset of tests that are fast and known to PASS. gdb_test_exp = ("TESTS= " "gdb.base/break-always.exp " "gdb.base/break-caller-line.exp " "gdb.base/break-entry.exp " "gdb.base/break.exp " "gdb.base/break-fun-addr.exp " "gdb.base/break-include.exp " "gdb.base/break-inline.exp " "gdb.base/break-main-file-remove-fail.exp " "gdb.base/break-on-linker-gcd-function.exp " "gdb.base/breakpoint-in-ro-region.exp " "gdb.base/breakpoint-shadow.exp " "gdb.base/break-probes.exp " "gdb.gdb/unittest.exp " "gdb.server/unittest.exp ") Which will be run three times with make gdb-check, once without a target_board, once with native-gdbserver and once with native-extended- gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-arm64, fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64, opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-check because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561) I'll add a debian-i386 builder so there is more 32bit coverage. All are green now (with the latest change to remove break-idempotent) https://builder.sourceware.org/buildbot/#/builders?tags=gdb Question is if this is a good list, does it need more tests? And should it maybe be maintained in the binutils-gdb repo instead of in the builder repo? For example we could have a make check-gdb-ci target which does what the buildbot would do (and then the buildbot could just call that). Cheers, Mark ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 10:50 ` Mark Wielaard @ 2022-06-10 10:58 ` Luis Machado 2022-06-10 15:17 ` will schmidt 2022-06-10 15:54 ` Carl Love 0 siblings, 2 replies; 13+ messages in thread From: Luis Machado @ 2022-06-10 10:58 UTC (permalink / raw) To: Mark Wielaard, Carl Love, Will Schmidt; +Cc: gdb, buildbot On 6/10/22 11:50, Mark Wielaard wrote: > Hi, > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: >> On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: >>> On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: >>>> I always use gdb.base/break.exp as a good smoke test. If that one >>>> fails, then things >>>> are really broken. >>>> >>>> I think gdb.base/break*.exp should make a good smoke test list. >>>> We just need to exclude >>>> gdb.base/break-interp.exp, which is problematic on some targets. >>> >>> It never is just easy is it? :) You are right, I saw break- >>> interp.exp >>> fail... I tried to come up with a regexp but gave up given that it >>> has to go throug python first and then we don't know whether the >>> worker uses bash as /bin/sh so I just added them all (exclusing >>> break-interp.exp) as a list. >> >> Sigh, sorry, looks like gdb.base/break-unload-file.exp also sometimes >> fails. >> I have removed from the list. Hopefully the remaining list does >> actually pass. > > And it didn't :{ > Yeah. As expected, the GDB testsuite is a bit delicate when you start dealing with multiple architectures and modes. But I think this is good progress already. > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) > gdb.base/break-idempotent.exp under both native-gdbserver and native- > extended-gdbserver > https://builder.sourceware.org/buildbot/#builders/76/builds/446 > https://builder.sourceware.org/buildbot/#builders/85/builds/294 Those might be genuine issues. I'm cc-ing Carl and Will so they can chime in. > > So I have removed that one too from the list. > The CI test list now looks like: > > # Only a small subset of tests that are fast and known to PASS. > gdb_test_exp = ("TESTS= " > "gdb.base/break-always.exp " > "gdb.base/break-caller-line.exp " > "gdb.base/break-entry.exp " > "gdb.base/break.exp " > "gdb.base/break-fun-addr.exp " > "gdb.base/break-include.exp " > "gdb.base/break-inline.exp " > "gdb.base/break-main-file-remove-fail.exp " > "gdb.base/break-on-linker-gcd-function.exp " > "gdb.base/breakpoint-in-ro-region.exp " > "gdb.base/breakpoint-shadow.exp " > "gdb.base/break-probes.exp " > "gdb.gdb/unittest.exp " > "gdb.server/unittest.exp ") > > Which will be run three times with make gdb-check, once without a > target_board, once with native-gdbserver and once with native-extended- > gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian-arm64, > fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64, > opensuseleap-x86_64 (debian-armhf only does a build, no make gdb-check > because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561) > I'll add a debian-i386 builder so there is more 32bit coverage. > > All are green now (with the latest change to remove break-idempotent) > https://builder.sourceware.org/buildbot/#/builders?tags=gdb Looks nice! :-) > > Question is if this is a good list, does it need more tests? And should > it maybe be maintained in the binutils-gdb repo instead of in the > builder repo? > > For example we could have a make check-gdb-ci target which does what > the buildbot would do (and then the buildbot could just call that). Having a new check-gdb target that only does minimal smoke tests should be easy to do. Once we determine a subset of critical tests, we can put something together if folks think it is a good idea. I like it, as it make it easier to deal with stability issues of GDB's testsuite across different targets. > > Cheers, > > Mark ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 10:58 ` Luis Machado @ 2022-06-10 15:17 ` will schmidt 2022-06-10 15:54 ` Carl Love 1 sibling, 0 replies; 13+ messages in thread From: will schmidt @ 2022-06-10 15:17 UTC (permalink / raw) To: Luis Machado, Mark Wielaard, Carl Love; +Cc: gdb, buildbot On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote: > On 6/10/22 11:50, Mark Wielaard wrote: > > Hi, > > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > > > > I always use gdb.base/break.exp as a good smoke test. If that > > > > > one > > > > > fails, then things > > > > > are really broken. > > > > > > > > > > I think gdb.base/break*.exp should make a good smoke test > > > > > list. > > > > > We just need to exclude > > > > > gdb.base/break-interp.exp, which is problematic on some > > > > > targets. > > > > > > > > It never is just easy is it? :) You are right, I saw break- > > > > interp.exp > > > > fail... I tried to come up with a regexp but gave up given > > > > that it > > > > has to go throug python first and then we don't know whether > > > > the > > > > worker uses bash as /bin/sh so I just added them all (exclusing > > > > break-interp.exp) as a list. > > > > > > Sigh, sorry, looks like gdb.base/break-unload-file.exp also > > > sometimes > > > fails. > > > I have removed from the list. Hopefully the remaining list does > > > actually pass. > > > > And it didn't :{ > > > > Yeah. As expected, the GDB testsuite is a bit delicate when you start > dealing with > multiple architectures and modes. But I think this is good progress > already. > > > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) > > gdb.base/break-idempotent.exp under both native-gdbserver and > > native- > > extended-gdbserver > > https://builder.sourceware.org/buildbot/#builders/76/builds/446 > > https://builder.sourceware.org/buildbot/#builders/85/builds/294 > > Those might be genuine issues. I'm cc-ing Carl and Will so they can > chime in. RROR: no fileid for debian-ppc64-builder UNRESOLVED: gdb.base/break-idempotent.exp: pie=nopie: always_inserted=off: watch: gdb_breakpoint: set breakpoint at main (timeout) I've looked briefly at the logs. I recall one of those tests having a FAIL with one of the test permutations, but not an UNRESOLVED. We'll need to poke further. has_hw_wp_support: Default, hardware watchpoint not deteced I see a typo in there somewhere... :-) > > > So I have removed that one too from the list. > > The CI test list now looks like: > > > > # Only a small subset of tests that are fast and known to PASS. > > gdb_test_exp = ("TESTS= " > > "gdb.base/break-always.exp " > > "gdb.base/break-caller-line.exp " > > "gdb.base/break-entry.exp " > > "gdb.base/break.exp " > > "gdb.base/break-fun-addr.exp " > > "gdb.base/break-include.exp " > > "gdb.base/break-inline.exp " > > "gdb.base/break-main-file-remove-fail.exp " > > "gdb.base/break-on-linker-gcd-function.exp " > > "gdb.base/breakpoint-in-ro-region.exp " > > "gdb.base/breakpoint-shadow.exp " > > "gdb.base/break-probes.exp " > > "gdb.gdb/unittest.exp " > > "gdb.server/unittest.exp ") > > > > Which will be run three times with make gdb-check, once without a > > target_board, once with native-gdbserver and once with native- > > extended- > > gdbserver on centos-x86_64, fedora-x86_64, debian-armhf, debian- > > arm64, > > fedora-s390x, debian-ppc64, fedora-ppc64le, opensusetw-x86_64, > > opensuseleap-x86_64 (debian-armhf only does a build, no make gdb- > > check > > because of https://sourceware.org/bugzilla/show_bug.cgi?id=28561) > > I'll add a debian-i386 builder so there is more 32bit coverage. > > > > All are green now (with the latest change to remove break- > > idempotent) > > https://builder.sourceware.org/buildbot/#/builders?tags=gdb > > Looks nice! :-) > > > Question is if this is a good list, does it need more tests? And > > should > > it maybe be maintained in the binutils-gdb repo instead of in the > > builder repo? > > > > For example we could have a make check-gdb-ci target which does > > what > > the buildbot would do (and then the buildbot could just call that). > > Having a new check-gdb target that only does minimal smoke tests > should > be easy to do. Once we determine a subset of critical tests, we can > put something > together if folks think it is a good idea. I like it, as it make it > easier to deal with > stability issues of GDB's testsuite across different targets. > > > Cheers, > > > > Mark ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 10:58 ` Luis Machado 2022-06-10 15:17 ` will schmidt @ 2022-06-10 15:54 ` Carl Love 2022-06-10 19:11 ` Mark Wielaard 1 sibling, 1 reply; 13+ messages in thread From: Carl Love @ 2022-06-10 15:54 UTC (permalink / raw) To: Luis Machado, Mark Wielaard, Will Schmidt; +Cc: gdb, buildbot On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote: > On 6/10/22 11:50, Mark Wielaard wrote: > > Hi, > > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > > > > I always use gdb.base/break.exp as a good smoke test. If that > > > > > one > > > > > fails, then things > > > > > are really broken. > > > > > > > > > > I think gdb.base/break*.exp should make a good smoke test > > > > > list. > > > > > We just need to exclude > > > > > gdb.base/break-interp.exp, which is problematic on some > > > > > targets. Trying to to understand how you are running the test. I think you are running these as remote tests, i.e. machine A requests that a remote machine B run the test via gdbserver, correct? I tried running the tests naitively, i.e. on a Power 10 RHEL 9 system I ran: make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break- interp.exp ' The gdb summary is: === gdb Summary === # of expected passes 702 # of expected failures 20 > > > > > > > > It never is just easy is it? :) You are right, I saw break- > > > > interp.exp > > > > fail... I tried to come up with a regexp but gave up given > > > > that it > > > > has to go throug python first and then we don't know whether > > > > the > > > > worker uses bash as /bin/sh so I just added them all (exclusing > > > > break-interp.exp) as a list. > > > > > > Sigh, sorry, looks like gdb.base/break-unload-file.exp also > > > sometimes > > > fails. > > > I have removed from the list. Hopefully the remaining list does > > > actually pass. I ran gdb.base/break-unload-file.exp on the same Power 10 RHEL 9 system. make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break-unload- file.exp ' The gdb summary is: === gdb Summary === # of expected passes 36 So it doesn't appear the tests are fundamentally broken. We have only played around with running the gdb testsuite remotely a little. Specifically, we were working on documenting how it is done. We see a number of additional issues running remotely versus natively but at this point have not had the time/bandwidth to dig into them > > > > And it didn't :{ > > > > Yeah. As expected, the GDB testsuite is a bit delicate when you start > dealing with > multiple architectures and modes. But I think this is good progress > already. > > > Both debian-ppc64 and fedora-ppc64le failed (UNRESOLVED) > > gdb.base/break-idempotent.exp under both native-gdbserver and > > native- > > extended-gdbserver Running that test natively, i.e. make check RUNTESTFLAGS='GDB=/home/carll/bin/gdb gdb.base/break- idempotent.exp ' generates a number of errors: ERROR: breakpoints not deleted ERROR: no fileid for ltcd97-lp2 ERROR: Couldn't send delete breakpoints to GDB. ERROR: can't read "gdb_spawn_id": no such variable while executing "expect { -i 1000 -timeout 100 -re ".*A problem internal to GDB has been detected" { fail "$message (GDB internal error)" gdb_internal_erro..." ("uplevel" body line 1) invoked from within "uplevel $body" TCL READ VARNAME can't read "gdb_spa The gdb summary is: === gdb Summary === # of expected passes 35 # of unresolved testcases 3 > > https://builder.sourceware.org/buildbot/#builders/76/builds/446 > > https://builder.sourceware.org/buildbot/#builders/85/builds/294 > > Those might be genuine issues. I'm cc-ing Carl and Will so they can > chime in. > Need to work on getting this one fixed for a native run before worrying about running it remotely. I will add this to the list of gdb issues that needs work. Carl ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 15:54 ` Carl Love @ 2022-06-10 19:11 ` Mark Wielaard 2022-06-10 20:11 ` Carl Love 2022-06-29 15:58 ` Carl Love 0 siblings, 2 replies; 13+ messages in thread From: Mark Wielaard @ 2022-06-10 19:11 UTC (permalink / raw) To: Carl Love; +Cc: Luis Machado, Will Schmidt, gdb, buildbot Hi Carl, On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote: > On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote: > > On 6/10/22 11:50, Mark Wielaard wrote: > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard wrote: > > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado wrote: > > > > > > I always use gdb.base/break.exp as a good smoke test. If that > > > > > > one > > > > > > fails, then things > > > > > > are really broken. > > > > > > > > > > > > I think gdb.base/break*.exp should make a good smoke test > > > > > > list. > > > > > > We just need to exclude > > > > > > gdb.base/break-interp.exp, which is problematic on some > > > > > > targets. > > Trying to to understand how you are running the test. I think you are > running these as remote tests, i.e. machine A requests that a remote > machine B run the test via gdbserver, correct? Yes, but machine A and B are the same one in this case. The tests are run three times. Once with just gdb, and then with a target-board set to either native-gdbserver or native-extended-gdbserver. The buildbot log stdio should show you the exact settings. https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp gdb.base/break-probes.exp gdb.gdb/unittest.exp gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native-gdbserver"' So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that tells the test to run against the native-gdbserver. Cheers, Mark ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 19:11 ` Mark Wielaard @ 2022-06-10 20:11 ` Carl Love 2022-06-10 22:23 ` Frank Ch. Eigler 2022-06-29 15:58 ` Carl Love 1 sibling, 1 reply; 13+ messages in thread From: Carl Love @ 2022-06-10 20:11 UTC (permalink / raw) To: Mark Wielaard; +Cc: Luis Machado, Will Schmidt, gdb, buildbot Mark: On Fri, 2022-06-10 at 21:11 +0200, Mark Wielaard wrote: > Hi Carl, > > On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote: > > On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote: > > > On 6/10/22 11:50, Mark Wielaard wrote: > > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > > > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard > > > > > wrote: > > > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado > > > > > > wrote: > > > > > > > I always use gdb.base/break.exp as a good smoke test. If > > > > > > > that > > > > > > > one > > > > > > > fails, then things > > > > > > > are really broken. > > > > > > > > > > > > > > I think gdb.base/break*.exp should make a good smoke test > > > > > > > list. > > > > > > > We just need to exclude > > > > > > > gdb.base/break-interp.exp, which is problematic on some > > > > > > > targets. > > > > Trying to to understand how you are running the test. I think you > > are > > running these as remote tests, i.e. machine A requests that a > > remote > > machine B run the test via gdbserver, correct? > > Yes, but machine A and B are the same one in this case. The tests > are > run three times. Once with just gdb, and then with a target-board set > to either native-gdbserver or native-extended-gdbserver. > > The buildbot log stdio should show you the exact settings. > > https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio > > make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break- > caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp > gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp > gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break- > main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp > gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp > gdb.base/break-probes.exp gdb.gdb/unittest.exp > gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native- > gdbserver"' > > So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that > tells > the test to run against the native-gdbserver. OK, thanks that will be helpful for testing. So, I have been trying to isolate where the messages ERROR: no fileid for ltcd97-lp2 ERROR: Couldn't send delete breakpoints to GDB. ERROR: can't read "gdb_spawn_id": no such variable Looks like a couple of the error messages are coming from proc default_gdb_exit. The routine is called as part of the check in the original test program. The break-idempotent.exp calls skip_hw_watchpoint_tests which calls has_hw_wp_support which calls gdb_exit which calls proc default_gdb_exit. In default_gdb_exit we have: if ![is_remote host] { remote_close host <- generates ERROR: no fileid for ltcd97-lp2 } unset gdb_spawn_id <- generates ERROR: can't read "gdb_spawn_id": no such variable unset ::gdb_tty_name unset inferior_spawn_id So, now to figure out why those statements are failing....??? Carl ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 20:11 ` Carl Love @ 2022-06-10 22:23 ` Frank Ch. Eigler 0 siblings, 0 replies; 13+ messages in thread From: Frank Ch. Eigler @ 2022-06-10 22:23 UTC (permalink / raw) To: Carl Love; +Cc: Mark Wielaard, gdb, buildbot, Will Schmidt, Luis Machado Hi - As an aside: now that these triplicated test results are starting to flow properly into bunsen, you can try having it do some comparisons for you: https://tinyurl.com/bdz4xyv8 to show the list of recent test/build runs from this batch. Can click on any subset's checkbox, hit "diff", and from there go to the respective runs' logfile extracts. - FChE ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-10 19:11 ` Mark Wielaard 2022-06-10 20:11 ` Carl Love @ 2022-06-29 15:58 ` Carl Love 2022-06-29 22:42 ` Mark Wielaard 1 sibling, 1 reply; 13+ messages in thread From: Carl Love @ 2022-06-29 15:58 UTC (permalink / raw) To: Mark Wielaard, cel; +Cc: Luis Machado, Will Schmidt, gdb, buildbot Mark: The test was failing the same way when running locally. It was a test issue related to checking for hardware watchpoint support on PowerPC. I posted a patch to fix it. Andrew Burgess merged a similar patch that he did to fix the test with my patch and committed the fix. The test is now running correctly locally. Note, I have not tested it in the remote configuration as you mentioned. Here is the commit for the fix: commit 13f72372413400410aaa94b7f0e2ff7de663fdcb Author: Carl Love <cel@us.ibm.com> Date: Wed Jun 22 16:14:20 2022 +0100 gdb/testsuite: fix gdb.base/break-idempotent.exp on ppc When running the gdb.base/break-idempotent.exp test on ppc, I was seeing some test failures (or rather errors), that looked like this: (gdb) watch local Hardware watchpoint 2: local has_hw_wp_support: Hardware watchpoint detected ERROR: no fileid for gcc2-power8 ERROR: Couldn't send delete breakpoints to GDB. ERROR OCCURED: can't read "gdb_spawn_id": no such variable while executing ... Please let me know if you see any additional issues with the test. Thanks. Carl On Fri, 2022-06-10 at 21:11 +0200, Mark Wielaard wrote: > Hi Carl, > > On Fri, Jun 10, 2022 at 08:54:00AM -0700, Carl Love wrote: > > On Fri, 2022-06-10 at 11:58 +0100, Luis Machado wrote: > > > On 6/10/22 11:50, Mark Wielaard wrote: > > > > On Fri, 2022-06-10 at 01:21 +0200, Mark Wielaard wrote: > > > > > On Fri, Jun 10, 2022 at 01:09:19AM +0200, Mark Wielaard > > > > > wrote: > > > > > > On Thu, Jun 09, 2022 at 10:37:58AM +0100, Luis Machado > > > > > > wrote: > > > > > > > I always use gdb.base/break.exp as a good smoke test. If > > > > > > > that > > > > > > > one > > > > > > > fails, then things > > > > > > > are really broken. > > > > > > > > > > > > > > I think gdb.base/break*.exp should make a good smoke test > > > > > > > list. > > > > > > > We just need to exclude > > > > > > > gdb.base/break-interp.exp, which is problematic on some > > > > > > > targets. > > > > Trying to to understand how you are running the test. I think you > > are > > running these as remote tests, i.e. machine A requests that a > > remote > > machine B run the test via gdbserver, correct? > > Yes, but machine A and B are the same one in this case. The tests > are > run three times. Once with just gdb, and then with a target-board set > to either native-gdbserver or native-extended-gdbserver. > > The buildbot log stdio should show you the exact settings. > > https://builder.sourceware.org/buildbot/#/builders/76/builds/446/steps/12/logs/stdio > > make -j4 check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break- > caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp > gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp > gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break- > main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp > gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp > gdb.base/break-probes.exp gdb.gdb/unittest.exp > gdb.server/unittest.exp ' 'RUNTESTFLAGS="--target_board=native- > gdbserver"' > > So it is the RUNTESTFLAGS="--target_board=native-gdbserver" that > tells > the test to run against the native-gdbserver. > > Cheers, > > Mark > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [builder] gdb_check_step: remove gdb.gdb/selftest.exp 2022-06-29 15:58 ` Carl Love @ 2022-06-29 22:42 ` Mark Wielaard 0 siblings, 0 replies; 13+ messages in thread From: Mark Wielaard @ 2022-06-29 22:42 UTC (permalink / raw) To: Carl Love; +Cc: Luis Machado, Will Schmidt, gdb, buildbot [-- Attachment #1: Type: text/plain, Size: 1347 bytes --] Hi Carl, On Wed, Jun 29, 2022 at 08:58:43AM -0700, Carl Love wrote: > The test was failing the same way when running locally. It was a test > issue related to checking for hardware watchpoint support on PowerPC. I > posted a patch to fix it. Andrew Burgess merged a similar patch that > he did to fix the test with my patch and committed the fix. The test > is now running correctly locally. Note, I have not tested it in the > remote configuration as you mentioned. Here is the commit for the > fix: > > commit 13f72372413400410aaa94b7f0e2ff7de663fdcb > Author: Carl Love <cel@us.ibm.com> > Date: Wed Jun 22 16:14:20 2022 +0100 > > gdb/testsuite: fix gdb.base/break-idempotent.exp on ppc > > When running the gdb.base/break-idempotent.exp test on ppc, I was > seeing some test failures (or rather errors), that looked like > this: > > (gdb) watch local > Hardware watchpoint 2: local > > has_hw_wp_support: Hardware watchpoint detected > ERROR: no fileid for gcc2-power8 > ERROR: Couldn't send delete breakpoints to GDB. > ERROR OCCURED: can't read "gdb_spawn_id": no such variable > while executing > ... > > Please let me know if you see any additional issues with the test. Lets add it back to the buildbot tests and see if anything breaks. Cheers, Mark [-- Attachment #2: 0001-gdb_test_exp-Add-gdb.base-break-idempotent.exp-back.patch --] [-- Type: text/x-diff, Size: 844 bytes --] From a1f7675ab2fa8a4c987d0e7f1fb53de564cf9ed9 Mon Sep 17 00:00:00 2001 From: Mark Wielaard <mark@klomp.org> Date: Thu, 30 Jun 2022 00:40:08 +0200 Subject: [PATCH] gdb_test_exp: Add gdb.base/break-idempotent.exp back This should now work on all arches --- builder/master.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/builder/master.cfg b/builder/master.cfg index 219f88d..729ec37 100644 --- a/builder/master.cfg +++ b/builder/master.cfg @@ -1820,6 +1820,7 @@ gdb_test_exp = ("TESTS= " "gdb.base/break-entry.exp " "gdb.base/break.exp " "gdb.base/break-fun-addr.exp " + "gdb.base/break-idempotent.exp " "gdb.base/break-include.exp " "gdb.base/break-inline.exp " "gdb.base/break-main-file-remove-fail.exp " -- 2.30.2 ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2022-06-29 22:42 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-08 18:51 [builder] gdb_check_step: remove gdb.gdb/selftest.exp Mark Wielaard 2022-06-09 9:37 ` Luis Machado 2022-06-09 23:09 ` Mark Wielaard 2022-06-09 23:21 ` Mark Wielaard 2022-06-10 10:50 ` Mark Wielaard 2022-06-10 10:58 ` Luis Machado 2022-06-10 15:17 ` will schmidt 2022-06-10 15:54 ` Carl Love 2022-06-10 19:11 ` Mark Wielaard 2022-06-10 20:11 ` Carl Love 2022-06-10 22:23 ` Frank Ch. Eigler 2022-06-29 15:58 ` Carl Love 2022-06-29 22:42 ` Mark Wielaard
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).