Labelling of regressions in Bugzilla

Labelling of regressions in Bugzilla

[Jonathan Wakely]

On IRC we've been discussing some changes to Bugzilla that would give
a bit more structure to how we label and process regressions.

Currently we add something like "[9/10/11/12 Regression]" to the start
of the summary, and then edit that when it's fixed on a branch, when
forking a new release branch from trunk, and when the oldest branch is
closed and becomes unmaintained. Finding active regressions means a
free-text search in the summary field.

On IRC we discussed adding a new custom field to record which branches
a regression affects. This could be like the
known-towork/known-to-fail fields, so only allow predefined values,
and allow searching by "all of" and by "any of". The possible values
for the field would only be the major releases, not all the minor
releases and snapshots and vendor branches that are in the
known-to-work field. So just 4.9, 5, 6, 7 etc. not 4.9.4, 5.0, 5.1.0,
5.1.1 etc.

When a new branch is forked from trunk before a release all bugs that
have "trunk" in the regression field would automatically get "12"
added (or if we already used "12" instead of "trunk" they'd get "13"
added, either way would work).

Unlike the current system, we wouldn't need to remove closed branches
from the regressions field. We do that today so the Summary field
doesn't get cluttered with old branch info, but if it's in a separate
field keeping the old data present is valuable. We would only remove a
branch from that field when the regression is fixed on the branch. We
would still be able to search for regressions in active branches
easily, but it would also be possible to see at a glance that a given
regression was present on the gcc-8 branch and never fixed there. This
would also help vendors who maintain older branches, as the
information that a regression affected an old branch would not be
wiped out of the summary when the branch reaches EOL.

Jakub also suggested it would be nice to be able to enter a revision
into a "regressed at" field and have it auto-populate the regressions
list with all branches that the commit is present in. (Ideally any of
SVN rNNNN numbers, or git revisions, or gcc-descr rNN-NNN strings
could be used there). That would be useful when we bisect the
regression and find where it started.

Iain pointed out a drawback of not having the regression info in the
Summary. Currently it does draw your attention when looking at the
results of a bugzilla search. Andrew noted that bug aliases are
automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
shows its alias "(c++core-issues)". Maybe we could do that for
regressions (for the active branches only, so the result would be
similar to what we have today).

Thoughts? Objections? Better ideas?

Re: Labelling of regressions in Bugzilla

[Richard Earnshaw]

On 15/12/2021 11:39, Jonathan Wakely via Gcc wrote:
> On IRC we've been discussing some changes to Bugzilla that would give
> a bit more structure to how we label and process regressions.
> 
> Currently we add something like "[9/10/11/12 Regression]" to the start
> of the summary, and then edit that when it's fixed on a branch, when
> forking a new release branch from trunk, and when the oldest branch is
> closed and becomes unmaintained. Finding active regressions means a
> free-text search in the summary field.
> 
> On IRC we discussed adding a new custom field to record which branches
> a regression affects. This could be like the
> known-towork/known-to-fail fields, so only allow predefined values,
> and allow searching by "all of" and by "any of". The possible values
> for the field would only be the major releases, not all the minor
> releases and snapshots and vendor branches that are in the
> known-to-work field. So just 4.9, 5, 6, 7 etc. not 4.9.4, 5.0, 5.1.0,
> 5.1.1 etc.
> 
> When a new branch is forked from trunk before a release all bugs that
> have "trunk" in the regression field would automatically get "12"
> added (or if we already used "12" instead of "trunk" they'd get "13"
> added, either way would work).
> 
> Unlike the current system, we wouldn't need to remove closed branches
> from the regressions field. We do that today so the Summary field
> doesn't get cluttered with old branch info, but if it's in a separate
> field keeping the old data present is valuable. We would only remove a
> branch from that field when the regression is fixed on the branch. We
> would still be able to search for regressions in active branches
> easily, but it would also be possible to see at a glance that a given
> regression was present on the gcc-8 branch and never fixed there. This
> would also help vendors who maintain older branches, as the
> information that a regression affected an old branch would not be
> wiped out of the summary when the branch reaches EOL.
> 
> Jakub also suggested it would be nice to be able to enter a revision
> into a "regressed at" field and have it auto-populate the regressions
> list with all branches that the commit is present in. (Ideally any of
> SVN rNNNN numbers, or git revisions, or gcc-descr rNN-NNN strings
> could be used there). That would be useful when we bisect the
> regression and find where it started.
> 
> Iain pointed out a drawback of not having the regression info in the
> Summary. Currently it does draw your attention when looking at the
> results of a bugzilla search. Andrew noted that bug aliases are
> automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
> shows its alias "(c++core-issues)". Maybe we could do that for
> regressions (for the active branches only, so the result would be
> similar to what we have today).
> 
> Thoughts? Objections? Better ideas?
> 

My immediate thought (since I tend to dislike deleting history) is why 
not have two fields?  One listing all the release branches where this 
has occurred and another for where it has now been fixed.  That way you 
can see quickly whether the regression has ever affected some versions 
of a release.  Something we lack today with the single fixed in field is 
the ability to track exactly which dot releases of each branch contained 
the fix for a regression.

Other than that, I have no other concerns at the moment.

Re: Labelling of regressions in Bugzilla

[Tobias Burnus]

On 15.12.21 12:39, Jonathan Wakely via Gcc wrote:

> Iain pointed out a drawback of not having the regression info in the
> Summary. Currently it does draw your attention when looking at the
> results of a bugzilla search. Andrew noted that bug aliases are
> automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
> shows its alias "(c++core-issues)".
Wouldn't it be easier to click on the "[Change Columns]" button at the
bottom of the search result page and add the new field to the "Selected
Columns"? The known-to-(work/fail) columns are available, i.e. this
feature also works with custom fields.

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955

Re: Labelling of regressions in Bugzilla

[Jonathan Wakely]

On Wed, 15 Dec 2021 at 12:15, Richard Earnshaw wrote:
>
>
>
> On 15/12/2021 11:39, Jonathan Wakely via Gcc wrote:
> > On IRC we've been discussing some changes to Bugzilla that would give
> > a bit more structure to how we label and process regressions.
> >
> > Currently we add something like "[9/10/11/12 Regression]" to the start
> > of the summary, and then edit that when it's fixed on a branch, when
> > forking a new release branch from trunk, and when the oldest branch is
> > closed and becomes unmaintained. Finding active regressions means a
> > free-text search in the summary field.
> >
> > On IRC we discussed adding a new custom field to record which branches
> > a regression affects. This could be like the
> > known-towork/known-to-fail fields, so only allow predefined values,
> > and allow searching by "all of" and by "any of". The possible values
> > for the field would only be the major releases, not all the minor
> > releases and snapshots and vendor branches that are in the
> > known-to-work field. So just 4.9, 5, 6, 7 etc. not 4.9.4, 5.0, 5.1.0,
> > 5.1.1 etc.
> >
> > When a new branch is forked from trunk before a release all bugs that
> > have "trunk" in the regression field would automatically get "12"
> > added (or if we already used "12" instead of "trunk" they'd get "13"
> > added, either way would work).
> >
> > Unlike the current system, we wouldn't need to remove closed branches
> > from the regressions field. We do that today so the Summary field
> > doesn't get cluttered with old branch info, but if it's in a separate
> > field keeping the old data present is valuable. We would only remove a
> > branch from that field when the regression is fixed on the branch. We
> > would still be able to search for regressions in active branches
> > easily, but it would also be possible to see at a glance that a given
> > regression was present on the gcc-8 branch and never fixed there. This
> > would also help vendors who maintain older branches, as the
> > information that a regression affected an old branch would not be
> > wiped out of the summary when the branch reaches EOL.
> >
> > Jakub also suggested it would be nice to be able to enter a revision
> > into a "regressed at" field and have it auto-populate the regressions
> > list with all branches that the commit is present in. (Ideally any of
> > SVN rNNNN numbers, or git revisions, or gcc-descr rNN-NNN strings
> > could be used there). That would be useful when we bisect the
> > regression and find where it started.
> >
> > Iain pointed out a drawback of not having the regression info in the
> > Summary. Currently it does draw your attention when looking at the
> > results of a bugzilla search. Andrew noted that bug aliases are
> > automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
> > shows its alias "(c++core-issues)". Maybe we could do that for
> > regressions (for the active branches only, so the result would be
> > similar to what we have today).
> >
> > Thoughts? Objections? Better ideas?
> >
>
> My immediate thought (since I tend to dislike deleting history) is why
> not have two fields?  One listing all the release branches where this
> has occurred and another for where it has now been fixed.  That way you
> can see quickly whether the regression has ever affected some versions
> of a release.

The Known To Fail field could be used for that, if we were more
diligent about using it when fixes are backported to a branch.

> Something we lack today with the single fixed in field is
> the ability to track exactly which dot releases of each branch contained
> the fix for a regression.

Yes, I use Target Milestone for the oldest dot release and then add
something like "Fixed for 9.4, 10.3, 11.2 and trunk" in the comments.
That could be more structured.

>
> Other than that, I have no other concerns at the moment.

Thanks for the feedback!

I think my concern with the idea is that when we make this info more
structured, there's a danger of giving ourselves more work to do
pushing buttons just for the sake of it. I think it's OK to make it
optional to use these fields, so that those who want to record the
rich info (e.g. me, maybe you, and Andrew Pinski when he goes on a
bugzilla tidying mission :-) can do so. The danger then is that using
them for search isn't reliable because the fields aren't used
consistently.

I think I'd still rather than the option to use them. At least for my
little corner of the project, I would aim to fill them in
consistently.

Re: Labelling of regressions in Bugzilla

[Jonathan Wakely]

On Wed, 15 Dec 2021 at 12:22, Tobias Burnus wrote:
>
> On 15.12.21 12:39, Jonathan Wakely via Gcc wrote:
>
> > Iain pointed out a drawback of not having the regression info in the
> > Summary. Currently it does draw your attention when looking at the
> > results of a bugzilla search. Andrew noted that bug aliases are
> > automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
> > shows its alias "(c++core-issues)".
> Wouldn't it be easier to click on the "[Change Columns]" button at the
> bottom of the search result page and add the new field to the "Selected
> Columns"? The known-to-(work/fail) columns are available, i.e. this
> feature also works with custom fields.

Yes, I'd be fine with that solution (thanks, for the reminder, I
should have mentioned that option in my initial mail).

If you reorder the "known to fail" column so it comes right before the
Summary column you would get a clear list of regressions shown before
the rest of the summary (and nothing in that column for
non-regressions).

A possible downside is that would show all the branches the regression
was on, including closed ones. Again, I'd be fine with that, but it's
a change from the info visible at a glance in the Summary today.

Re: Labelling of regressions in Bugzilla

[Iain Sandoe]

> On 15 Dec 2021, at 12:29, Jonathan Wakely via Gcc <gcc@gcc.gnu.org> wrote:
> 
> On Wed, 15 Dec 2021 at 12:22, Tobias Burnus wrote:
>> 
>> On 15.12.21 12:39, Jonathan Wakely via Gcc wrote:
>> 
>>> Iain pointed out a drawback of not having the regression info in the
>>> Summary. Currently it does draw your attention when looking at the
>>> results of a bugzilla search. Andrew noted that bug aliases are
>>> automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
>>> shows its alias "(c++core-issues)".
>> Wouldn't it be easier to click on the "[Change Columns]" button at the
>> bottom of the search result page and add the new field to the "Selected
>> Columns"? The known-to-(work/fail) columns are available, i.e. this
>> feature also works with custom fields.
> 
> Yes, I'd be fine with that solution (thanks, for the reminder, I
> should have mentioned that option in my initial mail).
> 
> If you reorder the "known to fail" column so it comes right before the
> Summary column you would get a clear list of regressions shown before
> the rest of the summary (and nothing in that column for
> non-regressions).
> 
> A possible downside is that would show all the branches the regression
> was on, including closed ones. Again, I'd be fine with that, but it's
> a change from the info visible at a glance in the Summary today.

I just tried this with my local search and it line-wraps the list so that it
does not matter too much about the number of branches reported.

However "known to fail” is not currently “regressed for” they have distinct
meanings (both of which are useful IMO).

Iain

Re: Labelling of regressions in Bugzilla

[Jonathan Wakely]

On Wed, 15 Dec 2021 at 12:43, Iain Sandoe wrote:
>
>
>
> > On 15 Dec 2021, at 12:29, Jonathan Wakely via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On Wed, 15 Dec 2021 at 12:22, Tobias Burnus wrote:
> >>
> >> On 15.12.21 12:39, Jonathan Wakely via Gcc wrote:
> >>
> >>> Iain pointed out a drawback of not having the regression info in the
> >>> Summary. Currently it does draw your attention when looking at the
> >>> results of a bugzilla search. Andrew noted that bug aliases are
> >>> automatically added to the summary, e.g. https://gcc.gnu.org/PR94404
> >>> shows its alias "(c++core-issues)".
> >> Wouldn't it be easier to click on the "[Change Columns]" button at the
> >> bottom of the search result page and add the new field to the "Selected
> >> Columns"? The known-to-(work/fail) columns are available, i.e. this
> >> feature also works with custom fields.
> >
> > Yes, I'd be fine with that solution (thanks, for the reminder, I
> > should have mentioned that option in my initial mail).
> >
> > If you reorder the "known to fail" column so it comes right before the
> > Summary column you would get a clear list of regressions shown before
> > the rest of the summary (and nothing in that column for
> > non-regressions).
> >
> > A possible downside is that would show all the branches the regression
> > was on, including closed ones. Again, I'd be fine with that, but it's
> > a change from the info visible at a glance in the Summary today.
>
> I just tried this with my local search and it line-wraps the list so that it
> does not matter too much about the number of branches reported.
>
> However "known to fail” is not currently “regressed for” they have distinct
> meanings (both of which are useful IMO).

Yes I didn't mean to imply that, sorry. I was just using the
known-to-fail field to test it, because we don't have the regressions
one to test with, and I sent a muddled reply.

What I should have said is that if you reorder the hypothetical new
field you would get a clear list of regressions.