Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[FX Coudert]

Hi,

I am finding it very hard to reliably compare test results and regressions with the very large number of gcc.dg/guality test failures that are apparently the new norm on x86_64-linux: more than one hundred on 13.1, and several hundreds on 14. Is there any on-going discussion about this?

I mean, from an almost-external point of view, these tests should probably be xfail'ed and a PR opened against them to reenable them.

FX

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Andreas Schwab]

On Jul 15 2023, FX Coudert via Gcc wrote:

> I am finding it very hard to reliably compare test results and regressions with the very large number of gcc.dg/guality test failures that are apparently the new norm on x86_64-linux: more than one hundred on 13.1, and several hundreds on 14. Is there any on-going discussion about this?

https://build.opensuse.org/package/live_build_log/devel:gcc:next/gcc14:gcc14-testresults/openSUSE_Tumbleweed/x86_64
shows 257 failures in */guality/*, which isn't unusual, I think.

> I mean, from an almost-external point of view, these tests should probably be xfail'ed and a PR opened against them to reenable them.

It could also be a bug in gdb.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Martin Jambor]

Hi,

On Sat, Jul 15 2023, FX Coudert via Gcc wrote:
> Hi,
>
> I am finding it very hard to reliably compare test results and regressions with the very large number of gcc.dg/guality test failures that are apparently the new norm on x86_64-linux: more than one hundred on 13.1, and several hundreds on 14. Is there any on-going discussion about this?
>
> I mean, from an almost-external point of view, these tests should probably be xfail'ed and a PR opened against them to reenable them.
>

As far as I understand it, the main problem is that it is not really
possible to XFAIL a test for one combination of options (say "-O2
-flto") and not others.

Martin

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Jakub Jelinek]

On Sat, Jul 15, 2023 at 10:58:40PM +0200, Martin Jambor wrote:
> Hi,
> 
> On Sat, Jul 15 2023, FX Coudert via Gcc wrote:
> > Hi,
> >
> > I am finding it very hard to reliably compare test results and regressions with the very large number of gcc.dg/guality test failures that are apparently the new norm on x86_64-linux: more than one hundred on 13.1, and several hundreds on 14. Is there any on-going discussion about this?
> >
> > I mean, from an almost-external point of view, these tests should probably be xfail'ed and a PR opened against them to reenable them.
> >
> 
> As far as I understand it, the main problem is that it is not really
> possible to XFAIL a test for one combination of options (say "-O2
> -flto") and not others.

It is far worse than that.  We have the target vs. different ISA options
vs. different -O? options vs. different versions of gdb matrix and maintaining
what cases from those are supposed to pass and what are expected to fail
is really hard.
Which is why people should just compare testsuite results from earlier run
on the same configuration to watch for regressions, especially in the
guality testsuite.

	Jakub

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[FX Coudert]

Hi,

> Which is why people should just compare testsuite results from earlier run
> on the same configuration to watch for regressions, especially in the
> guality testsuite.

All this gives the idea of a test framework that is too rigid, or tests that are too fragile. I mean, The accumulation of noise just decreases the value of the the test results.

FX

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Richard Sandiford]

Martin Jambor <mjambor@suse.cz> writes:
> Hi,
>
> On Sat, Jul 15 2023, FX Coudert via Gcc wrote:
>> Hi,
>>
>> I am finding it very hard to reliably compare test results and regressions with the very large number of gcc.dg/guality test failures that are apparently the new norm on x86_64-linux: more than one hundred on 13.1, and several hundreds on 14. Is there any on-going discussion about this?
>>
>> I mean, from an almost-external point of view, these tests should probably be xfail'ed and a PR opened against them to reenable them.
>>
>
> As far as I understand it, the main problem is that it is not really
> possible to XFAIL a test for one combination of options (say "-O2
> -flto") and not others.

FWIW, we do do that for aarch64*.  The principle is that if anyone
wants to work on improving debug quality, the XFAILs in the testsuite
are just as good a reference as bugzilla (unless the bugzilla PR has
specific analysis).

But it is still dependent on the default configuration and on the version
of GDB used.  It gives zero failures on my set-up, and seems to do the
same for some other gcc-testresults@ emails I've seen.  But it probably
won't for everyone.

I tend to update the XFAIL lists in stage 3/4, when no major changes
are expected.

Having the XFAILs doesn't really change the workflow.  It's still
necessary to do a before-after comparison.  But having fewer XPASSes &
FAILs makes it less likely that something gets missed.  It might even
encourage people to take guality regressions more seriously :)  Before,
there were so many XPASSes and FAILs in the "before" log that it was
easy think "well, what's one more?" or "this thing looks generally
broken, so I'll just ignore it".

Richard

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Mark Wielaard]

Hi,

On Sun, Jul 16, 2023 at 09:50:24AM +0200, FX Coudert via Gcc wrote:
> > Which is why people should just compare testsuite results from earlier run
> > on the same configuration to watch for regressions, especially in the
> > guality testsuite.
> 
> All this gives the idea of a test framework that is too rigid, or
> tests that are too fragile. I mean, The accumulation of noise just
> decreases the value of the the test results.

Note that one way to analyze whether tests are too fragile is to look
at the collected bunsen results:
https://builder.sourceware.org/testruns/?has_keyvalue_k=testrun.git_describe&has_keyvalue_op=glob&has_keyvalue_v=*gcc*

As far as I can see you are right that there are ~100 FAILs
(out of ~4700) in guality.exp on all arches/configs.

(CC Frank, the fedrawhide builder doesn't seem to include guality.exp,
do you know why?)

Cheers,

Mark

Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches

[Frank Ch. Eigler]

Hi -

Hi -

> (CC Frank, the fedrawhide builder doesn't seem to include guality.exp,
> do you know why?)

Good question.  The gcc.log testsuite file appears to be truncated as
it arrived into bunsen.  The test does appear to be running on the
bot.  Maybe the gcc's own dg* log processing scripts are messing
something up, before bunsen ever gets a chance to grab them. :(

- FChE