public inbox for bunsen@sourceware.org
 help / color / mirror / Atom feed
* [Bug bunsen/25091] [v1.0] analysis: which regressions are truly new?
  2019-01-01  0:00 [Bug bunsen/25091] New: [v1.0] analysis: which regressions are truly new? me at serhei dot io
@ 2019-01-01  0:00 ` me at serhei dot io
  2020-01-01  0:00 ` me at serhei dot io
  2022-09-16 17:15 ` [Bug bunsen/25091] " serhei at serhei dot io
  2 siblings, 0 replies; 4+ messages in thread
From: me at serhei dot io @ 2019-01-01  0:00 UTC (permalink / raw)
  To: bunsen

https://sourceware.org/bugzilla/show_bug.cgi?id=25091

--- Comment #1 from Serhei Makarov <me at serhei dot io> ---
Committed a basic version of +new_regressions, now that I'm satisfied it runs
in reasonable (roughly constant in the length of history) memory and finishes
in an extravagant (but not indefinitely-increasing) amount of time.

There may be some bugs to fix, not closing the PR yet.

Because the memory is constant as the algorithm computes *forward* over the
history, the next logical step is to cache the algorithm state and reuse it as
new testruns are added for more recent commits. That would solve the
extravagant-amount-of-time problem in practical use.

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Bug bunsen/25091] New: [v1.0] analysis: which regressions are truly new?
@ 2019-01-01  0:00 me at serhei dot io
  2019-01-01  0:00 ` [Bug bunsen/25091] " me at serhei dot io
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: me at serhei dot io @ 2019-01-01  0:00 UTC (permalink / raw)
  To: bunsen

https://sourceware.org/bugzilla/show_bug.cgi?id=25091

            Bug ID: 25091
           Summary: [v1.0] analysis: which regressions are truly new?
           Product: bunsen
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: bunsen
          Assignee: bunsen at sourceware dot org
          Reporter: me at serhei dot io
  Target Milestone: ---

Due to flaky tests flipping back and forth, the regression info generated by
+diff_commits is very noisy. In my presentation I described the next step as
'identifying and filtering out flaky testcases', but a more useful and simple
approach might be to search the entire history (by regression) and then
identify when a new regression *first* appears (is 'truly new').

This could be done either for the entire history, or for an n-month or n-commit
sliding window over the history (i.e. show a regression if it hasn't appeared
during previous n months/commits).

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Bug bunsen/25091] [v1.0] analysis: which regressions are truly new?
  2019-01-01  0:00 [Bug bunsen/25091] New: [v1.0] analysis: which regressions are truly new? me at serhei dot io
  2019-01-01  0:00 ` [Bug bunsen/25091] " me at serhei dot io
@ 2020-01-01  0:00 ` me at serhei dot io
  2022-09-16 17:15 ` [Bug bunsen/25091] " serhei at serhei dot io
  2 siblings, 0 replies; 4+ messages in thread
From: me at serhei dot io @ 2020-01-01  0:00 UTC (permalink / raw)
  To: bunsen

https://sourceware.org/bugzilla/show_bug.cgi?id=25091

--- Comment #2 from Serhei Makarov <me at serhei dot io> ---
Figuring out how to cache the +new_regressions analysis in a reasonable way.
The problem is that +new_regressions can be run with different key=* and
window_size/novelty_threshold=* arguments. So the cached data must be stored in
a way that allows recomputation if necessary.

My current thought is to store
- list of single_change: (name, subtest, outcome_pair)
- map of commit_pair -> list of key or '*'
- map of commit_pair -> list of (index of) single_change

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Bug bunsen/25091] analysis: which regressions are truly new?
  2019-01-01  0:00 [Bug bunsen/25091] New: [v1.0] analysis: which regressions are truly new? me at serhei dot io
  2019-01-01  0:00 ` [Bug bunsen/25091] " me at serhei dot io
  2020-01-01  0:00 ` me at serhei dot io
@ 2022-09-16 17:15 ` serhei at serhei dot io
  2 siblings, 0 replies; 4+ messages in thread
From: serhei at serhei dot io @ 2022-09-16 17:15 UTC (permalink / raw)
  To: bunsen

https://sourceware.org/bugzilla/show_bug.cgi?id=25091

Serhei Makarov <serhei at serhei dot io> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|bunsen at sourceware dot org       |serhei at serhei dot io
            Summary|[v1.0] analysis: which      |analysis: which regressions
                   |regressions are truly new?  |are truly new?

--- Comment #3 from Serhei Makarov <serhei at serhei dot io> ---
I did a lot of experimentation with the regression-filtering view on the
prototype branch earlier this spring, but it's still far from perfect in terms
of accurate & easy-to-read results. Eventually I will try implementing
something similar to the prototype/2022:scripts_main/show_changes*.py scripts
on the new master branch, so this PR is still relevant going forward.

For the time being the 'best practice' for getting an overview of regressions
is to generate a set of grid views with R-show-testcases ( / show_testcases.py
), grab a coffee, and page through the results.

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-09-16 17:15 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-01  0:00 [Bug bunsen/25091] New: [v1.0] analysis: which regressions are truly new? me at serhei dot io
2019-01-01  0:00 ` [Bug bunsen/25091] " me at serhei dot io
2020-01-01  0:00 ` me at serhei dot io
2022-09-16 17:15 ` [Bug bunsen/25091] " serhei at serhei dot io

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).