public inbox for gdb-patches@sourceware.org
 help / color / mirror / Atom feed
From: Pedro Alves <palves@redhat.com>
To: David Edelsohn <dje.gcc@gmail.com>,
	Joel Brobecker <brobecker@adacore.com>
Cc: GDB Patches <gdb-patches@sourceware.org>
Subject: Re: GDB 7.9.90 available for testing
Date: Fri, 10 Jul 2015 14:33:00 -0000	[thread overview]
Message-ID: <559FD7C6.2060502@redhat.com> (raw)
In-Reply-To: <CAGWvnymxrXhGwC5oxXU_LpKhvY-KdsqgzEwihAV4ssS_Lekv6Q@mail.gmail.com>

On 07/10/2015 03:04 PM, David Edelsohn wrote:
> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>> I'm not certain if the baselines truly are accurate for all
>>> buildslaves, but it seems strange to create a release when the
>>> buildbot testsuite results show patches causing new failures.
>>
>> To me, you are saying the same thing, and I don't disagree with you.
>> I said I didn't know that the buildBots were showing regressions.
>> Of course I would have held the creation of the branch if I had
>> known about this. But I didn't, and so here we are. Now we all know,
>> and the only way forward is to look at those regressions, and decide
>> what to do. We can and will delay the release if we have to.
> 
> Joel,
> 
> We are agreeing.  I was trying to provide some additional information
> about interpretation of the buildbot status.
> 
> I am note two things about the buildbots:
> 
> 1) Their color-coded "regression status" apparently is a comparison of
> the testsuite between a "base" run and the current run. This is due to
> few or no targets have completely clean testsuite runs to consider
> "green".  Because there has been some adjustment and tweaking while
> buildbots were added, the first run was not necessarily the ideal one
> to choose as the "base" run, i.e., "regressions" may be due to changes
> in the measurements after the first "base" run, not new failing tests.

There's no single "base" run, actually.  The baseline is dynamically
adjusted at each build; it's a moving baseline, and it's per
test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
added to the baseline.  That means that if some test is racy, it'll
sometimes FAIL, and then a few builds later it'll PASS, at which point
the PASS is recorded in the baseline, and then a few builds again
later, the test FAIL again, and so the buildbot email report mentions
the regression against the baseline.  In sum, if a test goes
FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
get reports of regressions against the baseline for that racy test.

For each build, you can find the baseline file in the corresponding
git commit pointed at in the email report.  E.g., see the "baseline"
file here:

  http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b

The gdb.thread/ FAILs you see on AIX seem to fall in that category.
From the results, it looks to me that those are caused by the AIX port
not implementing schedlock correctly.  Is anyone from IBM available
to look at these?

The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
regressions, but new FAILs.  And they are really a test problem, not
a GDB bug.  They actually depend on compiler or debug info format
used, not system.

> 
> 2) Separate from the "regression" status, a quick inspection of some
> testsuite output in the buildbots show the introduction of new errors
> with recent commits.  Even if the overall regression status is not an
> accurate measure of the state of GDB on those targets, the change in
> regression status that is not monotonic reduction in regressions in
> preparation for a release is disappointing.
> 
> I'm not demanding STOP SHIP.  GDB may not necessarily be in a bad
> state for a release.  I don't know how this compares to the regression
> status of previous releases.
> 
> I hope that GDB developers will become more aware of the effects of
> their patches on multiple targets.

This is just a misunderstanding.

Thanks,
Pedro Alves

  reply	other threads:[~2015-07-10 14:33 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-07-09 22:34 David Edelsohn
2015-07-09 23:21 ` Joel Brobecker
2015-07-10  1:30   ` David Edelsohn
2015-07-10  3:43     ` Joel Brobecker
2015-07-10 14:04       ` David Edelsohn
2015-07-10 14:33         ` Pedro Alves [this message]
2015-07-10 14:56           ` David Edelsohn
2015-07-10 15:08             ` Pedro Alves
2015-07-10 15:25               ` David Edelsohn
2015-07-10 15:33                 ` Pedro Alves
2015-07-11 13:59           ` Doug Evans
2015-07-11 18:54             ` racy tests Pedro Alves
2015-07-14 16:05               ` Doug Evans
2015-07-14 20:26               ` Sergio Durigan Junior
  -- strict thread matches above, loose matches on Subject: below --
2015-07-06 20:29 GDB 7.9.90 available for testing Joel Brobecker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=559FD7C6.2060502@redhat.com \
    --to=palves@redhat.com \
    --cc=brobecker@adacore.com \
    --cc=dje.gcc@gmail.com \
    --cc=gdb-patches@sourceware.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).