Intermittent/non-reproducible gcc testsuite failures

Intermittent/non-reproducible gcc testsuite failures

[Michael Eager]

Hi --

I'm running the gcc test suite on powerpc-unknown-eabisim
on the trunk and I get results which are different from
one run to the next.  When I run the failing tests by
hand, all pass.  Mike Stein also has noted that some of
the tests are intermittent failures.

There are only a few test cases which I've noticed this
non-reproducible behavior:
    gcc.c-torture/execute/builtins/fprintf.c
    gcc.c-torture/execute/builtins/fputs.c
    gcc.c-torture/execute/builtins/printf.c
    gcc.c-torture/execute/20020406-1.c
    gcc.c-torture/execute/fprintf-1.c
    gcc.c-torture/execute/fprintf-chk-1.c
    gcc.c-torture/execute/printf-chk-1.c
    gcc.c-torture/execute/va-arg-21.c
    gcc.c-torture/execute/vfprintf-1.c
    gcc.c-torture/execute/vfprintf-chk-1.c
    gcc.dg/packed-array.c
    gcc.dg/matrix/matrix-2.c

Clearly, something is not initialized or is being overwritten.

I'm assuming that this is a problem only on powerpc
and perhaps only powerpc-eabi.

Does anyone have any suggestions on how to get one of
these tests to fail consistently, or a different approach
to finding the cause of the intermittent failures?

-- 
Michael Eager	 eager@eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077

Re: Intermittent/non-reproducible gcc testsuite failures

[Ben Elliston]

On Tue, 2009-04-07 at 23:45 -0700, Michael Eager wrote:

> I'm running the gcc test suite on powerpc-unknown-eabisim
> on the trunk and I get results which are different from
> one run to the next.  When I run the failing tests by
> hand, all pass.  Mike Stein also has noted that some of
> the tests are intermittent failures.

> Does anyone have any suggestions on how to get one of
> these tests to fail consistently, or a different approach
> to finding the cause of the intermittent failures?

There are two (or three?) sources of interittency here, and one of them
is the testsuite harness itself.  One thing you could do to try and
narrow it down is to run *just* the test cases you are seeing
intermittency in using RUNTESTFLAGS=foo.exp=bar.c a dozen times and see
if the results are stable.

Cheers, Ben

Re: Intermittent/non-reproducible gcc testsuite failures

[Dave Korn]

Michael Eager wrote:

> Does anyone have any suggestions on how to get one of
> these tests to fail consistently, or a different approach
> to finding the cause of the intermittent failures?

  Perhaps hack the testsuite to run the tests under gdb, setting a breakpoint
on abort() that causes it to dump core?  Then at least you'd be able to get a
backtrace and see some state post-mortem.

  Of course, being a Heisenbug, it'll probably stop appearing when run under a
debugger!

    cheers,
      DaveK

Re: Intermittent/non-reproducible gcc testsuite failures

[Michael Eager]

Dave Korn wrote:
> Michael Eager wrote:
> 
>> Does anyone have any suggestions on how to get one of
>> these tests to fail consistently, or a different approach
>> to finding the cause of the intermittent failures?
> 
>   Perhaps hack the testsuite to run the tests under gdb, setting a breakpoint
> on abort() that causes it to dump core?  Then at least you'd be able to get a
> backtrace and see some state post-mortem.

Thanks, Ben and Dave.  I'll give it a try.

>   Of course, being a Heisenbug, it'll probably stop appearing when run under a
> debugger!

Ah, yes.  Hiding in the shadows.


-- 
Michael Eager	 eager@eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077

Re: Intermittent/non-reproducible gcc testsuite failures

[Michael Eager]

Dave Korn wrote:
> Michael Eager wrote:
> 
>> Does anyone have any suggestions on how to get one of
>> these tests to fail consistently, or a different approach
>> to finding the cause of the intermittent failures?
> 
>   Perhaps hack the testsuite to run the tests under gdb, setting a breakpoint
> on abort() that causes it to dump core?  Then at least you'd be able to get a
> backtrace and see some state post-mortem.

I put together a dejagnu configuration to run tests
using powerpc-gdb.  The intermittent test cases pass
consistently.  Looks like a simulator bug.

I've filed a bug report in gdb:
http://sourceware.org/bugzilla/show_bug.cgi?id=10066

-- 
Michael Eager	 eager@eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077