* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
@ 2022-11-15 10:53 ` vries at gcc dot gnu.org
2022-11-15 11:49 ` vries at gcc dot gnu.org
` (8 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-15 10:53 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
Hmm, reading the definition of UNSUPPORTED:
...
There is no support for the tested case. This may mean that a conditional
feature of an operating system, or of a compiler, is not implemented. DejaGnu
also uses this message when a testing environment (often a “bare board” target)
lacks basic support for compiling or running the test case. For example, a test
for the system subroutine gethostname would never work on a target board
running only a boot monitor.
...
and seeing how like XFAIL it's focused on the environment, I start to wonder if
the role of UNTESTED was intended to be that it's to UNSUPPORTED what KFAIL is
to XFAIL.
In other words:
- xfail: fail due to problem in environment
- kfail: fail due to problem in gdb
- unsupported: not run due to missing optional feature in environment
- untested: not run due to missing optional feature in gdb
That way, each one implies one type of action to achieve a passing test-case:
- xfail: fix environment
- kfail: fix gdb
- unsupported: add missing feature to environment
- untested: build gdb with missing feature enabled
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
2022-11-15 10:53 ` [Bug testsuite/29778] " vries at gcc dot gnu.org
@ 2022-11-15 11:49 ` vries at gcc dot gnu.org
2022-11-15 14:35 ` simon.marchi at polymtl dot ca
` (7 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-15 11:49 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #2 from Tom de Vries <vries at gcc dot gnu.org> ---
And then there are the test-cases that use proc skip_ variants which largely
seem happy to skip only doing "verbose $msg" or just silently.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
2022-11-15 10:53 ` [Bug testsuite/29778] " vries at gcc dot gnu.org
2022-11-15 11:49 ` vries at gcc dot gnu.org
@ 2022-11-15 14:35 ` simon.marchi at polymtl dot ca
2022-11-18 15:33 ` vries at gcc dot gnu.org
` (6 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: simon.marchi at polymtl dot ca @ 2022-11-15 14:35 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
Simon Marchi <simon.marchi at polymtl dot ca> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |simon.marchi at polymtl dot ca
--- Comment #3 from Simon Marchi <simon.marchi at polymtl dot ca> ---
(In reply to Tom de Vries from comment #1)
> Hmm, reading the definition of UNSUPPORTED:
> ...
> There is no support for the tested case. This may mean that a conditional
> feature of an operating system, or of a compiler, is not implemented. DejaGnu
> also uses this message when a testing environment (often a “bare board”
> target)
> lacks basic support for compiling or running the test case. For example, a
> test
> for the system subroutine gethostname would never work on a target board
> running only a boot monitor.
> ...
> and seeing how like XFAIL it's focused on the environment, I start to wonder
> if the role of UNTESTED was intended to be that it's to UNSUPPORTED what
> KFAIL is to XFAIL.
>
> In other words:
> - xfail: fail due to problem in environment
> - kfail: fail due to problem in gdb
> - unsupported: not run due to missing optional feature in environment
> - untested: not run due to missing optional feature in gdb
I'm struggling a bit with the difference between unsupported and untested,
especially because unsupported sounds like a specific case of untested. And
except because of what the Dejagnu documentation says, unsupported really
sounds like it could be used for both "missing feature in environment" and
"missing feature in gdb". Using unsupported just for the former one seems
arbitrary.
> And then there are the test-cases that use proc skip_ variants which largely seem happy to skip only doing "verbose $msg" or just silently.
I think all tests should at least record at least one untested or unsupported,
so that there's a trace in gdb.sum.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (2 preceding siblings ...)
2022-11-15 14:35 ` simon.marchi at polymtl dot ca
@ 2022-11-18 15:33 ` vries at gcc dot gnu.org
2022-11-18 15:39 ` vries at gcc dot gnu.org
` (5 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-18 15:33 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #4 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Simon Marchi from comment #3)
> > And then there are the test-cases that use proc skip_ variants which largely seem happy to skip only doing "verbose $msg" or just silently.
>
> I think all tests should at least record at least one untested or
> unsupported, so that there's a trace in gdb.sum.
I think that's an ok principle, but once the volume grows the usefulness is
reduced.
I think it's fine to avoid generating noise. For instance, I would consider it
noise to record that a ppc64-specific test is not run on x86_64.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (3 preceding siblings ...)
2022-11-18 15:33 ` vries at gcc dot gnu.org
@ 2022-11-18 15:39 ` vries at gcc dot gnu.org
2022-11-18 15:41 ` vries at gcc dot gnu.org
` (4 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-18 15:39 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Simon Marchi from comment #3)
> > and seeing how like XFAIL it's focused on the environment, I start to wonder
> > if the role of UNTESTED was intended to be that it's to UNSUPPORTED what
> > KFAIL is to XFAIL.
> >
> > In other words:
> > - xfail: fail due to problem in environment
> > - kfail: fail due to problem in gdb
> > - unsupported: not run due to missing optional feature in environment
> > - untested: not run due to missing optional feature in gdb
>
> I'm struggling a bit with the difference between unsupported and untested,
> especially because unsupported sounds like a specific case of untested. And
> except because of what the Dejagnu documentation says, unsupported really
> sounds like it could be used for both "missing feature in environment" and
> "missing feature in gdb". Using unsupported just for the former one seems
> arbitrary.
>
I know some struggle with the difference between kfail and xfail and feel it's
arbitrary. For me the difference is clear: for a kfail you need to record a
gdb PR. For an xfail, you don't.
Between unsupported and untested, I agree what I proposed here can bee seen as
arbitrary, it's just an attempt to make sense of the available incomplete
information.
All I care about is not having to wonder each time whether to use untested or
unsupported, either by:
- having clear rules when to use them, or
- eliminated the use of one of them by generating an error, or
- simply mapping one onto the other.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (4 preceding siblings ...)
2022-11-18 15:39 ` vries at gcc dot gnu.org
@ 2022-11-18 15:41 ` vries at gcc dot gnu.org
2022-11-18 15:51 ` vries at gcc dot gnu.org
` (3 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-18 15:41 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #6 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #5)
> - simply mapping one onto the other.
Say:
...
proc unsupported { message } {
untested "unsupported: $message"
}
...
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (5 preceding siblings ...)
2022-11-18 15:41 ` vries at gcc dot gnu.org
@ 2022-11-18 15:51 ` vries at gcc dot gnu.org
2023-01-18 19:02 ` tromey at sourceware dot org
` (2 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: vries at gcc dot gnu.org @ 2022-11-18 15:51 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #7 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #4)
> (In reply to Simon Marchi from comment #3)
> > > And then there are the test-cases that use proc skip_ variants which largely seem happy to skip only doing "verbose $msg" or just silently.
> >
> > I think all tests should at least record at least one untested or
> > unsupported, so that there's a trace in gdb.sum.
>
> I think that's an ok principle, but once the volume grows the usefulness is
> reduced.
>
> I think it's fine to avoid generating noise. For instance, I would consider
> it noise to record that a ppc64-specific test is not run on x86_64.
And we could also avoid noise by emitting one untested/unsupported per skip
group. I don't need to know that gdb.server/*.exp is skipped for each
test-case, if gdbserver is unavailable.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (6 preceding siblings ...)
2022-11-18 15:51 ` vries at gcc dot gnu.org
@ 2023-01-18 19:02 ` tromey at sourceware dot org
2023-01-19 2:26 ` sam at gentoo dot org
2023-01-22 17:31 ` tromey at sourceware dot org
9 siblings, 0 replies; 11+ messages in thread
From: tromey at sourceware dot org @ 2023-01-18 19:02 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
Tom Tromey <tromey at sourceware dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |tromey at sourceware dot org
--- Comment #8 from Tom Tromey <tromey at sourceware dot org> ---
I wasn't aware of this bug when I wrote the "require" series.
I also stumbled over the weird text in dejagnu that made me
that "untested" was essentially useless. So, I changed require
to use unsupported.
On reflection though, I think what dejagnu says isn't super important.
We can just pick the one we like for gdb.
I tend to think the noise is fine. My reasoning is that these
procs give us a way to see not just that a test didn't run, but
also why it didn't run. Occasionally this reason could be surprising
and lead to some kind of bug fix. So my view is that the silent
"return"s in the test cases should be converted.
Centralizing these decisions with "require" or other "do task and
return on failure" procs makes this simpler to achieve.
(Here I was thinking of prepare_for_testing but it looks like
build_executable can call untested.)
One other way centralizing is nice is that if we decide to remove
the noise, we can just remove these calls and be done.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (7 preceding siblings ...)
2023-01-18 19:02 ` tromey at sourceware dot org
@ 2023-01-19 2:26 ` sam at gentoo dot org
2023-01-22 17:31 ` tromey at sourceware dot org
9 siblings, 0 replies; 11+ messages in thread
From: sam at gentoo dot org @ 2023-01-19 2:26 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
Sam James <sam at gentoo dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sam at gentoo dot org
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug testsuite/29778] [gdb/testsuite] Revise untested usage
2022-11-12 10:42 [Bug testsuite/29778] New: [gdb/testsuite] Revise untested usage vries at gcc dot gnu.org
` (8 preceding siblings ...)
2023-01-19 2:26 ` sam at gentoo dot org
@ 2023-01-22 17:31 ` tromey at sourceware dot org
9 siblings, 0 replies; 11+ messages in thread
From: tromey at sourceware dot org @ 2023-01-22 17:31 UTC (permalink / raw)
To: gdb-prs
https://sourceware.org/bugzilla/show_bug.cgi?id=29778
--- Comment #9 from Tom Tromey <tromey at sourceware dot org> ---
I found a case where the noisier output would be actively helpful.
For me, dw2-ranges.exp started silently doing nothing.
Investigating it:
/home/tromey/gdb/build/gdb/testsuite/../../../binutils-gdb/gdb/testsuite/gdb.dwarf2/dw2-ranges-3.c:
warning: STABS debugging information is obsolete and not supported anymore
So, a new compiler warning means we started skipping a test.
If every early return had some annotation that showed up in the .sum,
we could detect this scenario... not saying there's anything to do
about it, but it seems better to flag it an intentionally ignore
it than to have some tests simply silently stop running and never
really find out.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread