Unreliable test suites?

Unreliable test suites?

[Paul_Koning]

I'm doing some work on gdb and want to run the testsuites to confirm that I didn't mess it up.

The problem is that a number of them seem to be quite unreliable.  I've seen test runs where gdb.btrace/step.exp and/or stepi.exp have a pile of failures, but then when I rerun either just those tests, or the whole suite, they pass.

Since I haven't a clue how the reverse execution stuff works, I don't know if this is expected.  It seems strange.  I also don't know what to do about it if it's not supposed to be like that. For now, I'm just running things a couple of times, and if they pass once, I call it good enough.

	paul

RE: Unreliable test suites?

[Metzger, Markus T]

> -----Original Message-----
> From: gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] On
> Behalf Of Paul_Koning@Dell.com
> Sent: Wednesday, October 14, 2015 9:59 PM
> To: gdb@sourceware.org
> Subject: Unreliable test suites?

Hello Paul,

> The problem is that a number of them seem to be quite unreliable.  I've seen
> test runs where gdb.btrace/step.exp and/or stepi.exp have a pile of failures,
> but then when I rerun either just those tests, or the whole suite, they pass.

Those tests are not expected to fail.  They should either run if your processor
and OS support BTS or Intel PT or they should be skipped.

Would you please send me the gdb.log file for a failing run?

Thanks,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: Unreliable test suites?

[Doug Evans]

On Wed, Oct 14, 2015 at 12:58 PM,  <Paul_Koning@dell.com> wrote:
> I'm doing some work on gdb and want to run the testsuites to confirm that I didn't mess it up.
>
> The problem is that a number of them seem to be quite unreliable.  I've seen test runs where gdb.btrace/step.exp and/or stepi.exp have a pile of failures, but then when I rerun either just those tests, or the whole suite, they pass.
>
> Since I haven't a clue how the reverse execution stuff works, I don't know if this is expected.  It seems strange.  I also don't know what to do about it if it's not supposed to be like that. For now, I'm just running things a couple of times, and if they pass once, I call it good enough.

Those particular tests don't fail for me, even with check-parallel,
but then it could depend on the target.
OTOH several tests *are* flaky, especially under load.

What I normally do is for the failures,
run just those tests one at a time (to reduce load induced failures).
And do that in the before and after trees.
If they fail in the before tree too, then don't worry about it.

Re: Unreliable test suites?

[Paul_Koning]

> On Oct 26, 2015, at 1:36 AM, Doug Evans <xdje42@gmail.com> wrote:
> 
> On Wed, Oct 14, 2015 at 12:58 PM,  <Paul_Koning@dell.com> wrote:
>> I'm doing some work on gdb and want to run the testsuites to confirm that I didn't mess it up.
>> 
>> The problem is that a number of them seem to be quite unreliable.  I've seen test runs where gdb.btrace/step.exp and/or stepi.exp have a pile of failures, but then when I rerun either just those tests, or the whole suite, they pass.
>> 
>> Since I haven't a clue how the reverse execution stuff works, I don't know if this is expected.  It seems strange.  I also don't know what to do about it if it's not supposed to be like that. For now, I'm just running things a couple of times, and if they pass once, I call it good enough.
> 
> Those particular tests don't fail for me, even with check-parallel,
> but then it could depend on the target.
> OTOH several tests *are* flaky, especially under load.
> 
> What I normally do is for the failures,
> run just those tests one at a time (to reduce load induced failures).
> And do that in the before and after trees.
> If they fail in the before tree too, then don't worry about it.

Thanks.  I've been doing this sort of thing as well, and that helps.  Running just one test as a confirmation seems to be particularly effective.

	paul