Re: Bugzilla features

Re: Bugzilla features

[Matthew P. Barnson]

On Sun, 2001-11-18 at 17:04, Neil Booth wrote:
> Hi,
> 
> I work on GCC (gcc.gnu.org) and we currently use GNATS as our bug tracking
> system.  However, it is painful in many respects; just a few examples are:

  Funny thing you should mention that.  I went to a Linux convention
last week where I got to meet Bruce Perens (pretty cool!).  He mentioned
that savannah.gnu.org needed a real bug-tracking system, and asked if
I'd look into it.  There has been conversation this last week with the
core developers about making Bugzilla work under Savannah.  However,
that's not the be-all-and-end-all of Savannah work, it may be that a
complete rewrite is easier.

> Could someone who knows Bugzilla well possibly take some time to read the
> above mail and answer his points, back to this newsgroup? If you cc:
> gcc@gcc.gnu.org that would be great, but I'm happy to gather up the
> answers and relay them if that isn't done.

  See below.  I maintain documentation on Bugzilla, so I'm fairly
familiar with the warts and features, although I'm continually
pleasantly surprised every time I download the latest version :)


> In particular, we'd like to be able to do more administration and handling
> of bugs via e-mail (like the Debian bug system can).

  Not Bugzilla's strong point right now.  I have some patches pending
(sheesh, looking now, pending for over a year) which add some more
functionality to the email interface.  Some things don't work as
expected, but I anticipate a few days of hacking by a decent coder would
fix it up.


>  And would it be easy
> to migrate the existing GNATS database in some form, or would that
> information be lost?
$BUGZILLA_HOME/contrib/gnats2bz.pl .  May need some hacking to match
current database format.  I haven't used it in years.

-------------------------------------------------

Remainder is a response to the referenced questions:

1. Works in a generally mail-based system: bug submission by a gccbug
script, bugs sent to gcc-bugs / gcc-prs mailing lists, follow-ups that
go to a bug tracking system address stored with the bug and sent on the
gcc-prs, full audit trails of every change to a bug sent to gcc-prs.

  This is very do-able with Bugzilla, and is a relatively common setup. 
You would create a new Q/A contact for all new bugs in a product which
is actually an alias to a mailing list (the account should be disabled
in Bugzilla) held elsewhere.  No biggie.
  Someone would have to set up the filter on that account, but as I said
this is a very common thing to do with Bugzilla. 


2. Handles MIME in mail.

  You mean MIME attachments?  Yep, Bugzilla does it, but it doesn't
currently read MIME type really well.  If it's text, it shows up as
text, but anything that's binary is simply binary (Microsoft Word .doc
files and such) if downloaded from the web interface.  Also the binary
attachment will NOT be forwarded to the mailing list -- only the body
text will go as a comment.


3. Where files are attached in web submissions, they appear in mail as
MIME attachments, compressed if large, not as base64-encoded junk
needing manual decoding (as at present).
  As I mentioned before, rather than sucking up bandwidth by forwarding
the actual MIME attachment, bugmail sends a link to the web site and a
comment.  bugmail is *not* enabled at bugzilla.mozilla.org, however (due
to potential abuse reasons), so it's tough to try it out without setting
it up yourself.
  
4. All existing URLs for individual PRs get redirected to the
corresponding URLs in the new system.

  $BUGZILLA_HOME/contrib/gnats2bz.pl.  However, this perl script may
require some updating to match the new schema -- anybody can
confirm/deny this?  I used it once two years ago and it worked great,
but I haven't used it since.

5. Follow-ups sent to gcc-gnats for existing PRs get filed properly in
any new system.
  Hmm, mirroring a mailing list to the bugmail system?  That would be
problematical, but could be done with the right procmail filters in
place.  It's easier to say that followups to
"bugzilla-daemon@wherever.org" would go to bugmail, but not replies to
the mailing list.  Bugzilla's not a mailing list archive system, really.

6. No login required for browsing of bugs or submitting new bugs.

  Browsing is easily done.  Just don't set up bug security.  If you want
some bug security, it's just as trivial.  However, if someone wants to
create a bug, they have to set up an account.  This is a major
difference with it and gnats... Bugzilla, by default, is not an "open
bug receiver".  I have a patch to enable automatic bug creation upon
emailed bug submission, but I can tell you from experience that this is
an open door for Spammers to get a large target audience.  

7. Preferably possible to do all administration by mail (as with
debbugs).  (If more security than debbugs is wanted, plaintext passwords
(as at present) or digital signatures can be used to authenticate
changes.)

  Not available.

8. Either stores bug reports in a format such than non-experts can
readily locate and fix corruption, or a commitment must be available of
sysadmin time to fix corruption problems rapidly.  ISTR that Bugzilla
uses a database backend - which I'm not competent to fix corruption
problems with, and I had to deal with the last problem with PRs getting
lost since the main sysadmins weren't interested (see
http://sources.redhat.com/ml/overseers/2001-q2/msg00156.html and related
messages).

  Bugzilla is currently back-ended into MySQL (see
http://www.mozilla.org/projects/bugzilla/ ).  Dave Lawrence of Redhat
has done an oracle port, see http://bugzilla.redhat.com/.  Dawn Endico
made Bugzilla's XML DTD the industry-standard bug-tracking DTD (no
standardization committee, but most open bug-trackers support the format
now).  So you can do a mysql dump of data, export it via XML for
importation elsewhere, or export your existing data to an
easily-understood XML format for importation.
  Bugzilla includes a "sanity check" to find basic database issues with
data formats.  It's not comprehensive, however.  I recommend a good
daily backup strategy for any Bugzilla database, which is covered in
great detail at the mysql.com website.

9. The web interface should not treat the absence of bugs matching a
search as either an error condition or an opportunity for a silly
message;  it should simply state the absence of matching bugs.

  Bugzilla says "Zarro Boogs Found" on null searches.  Fixed easily by
doing "for i in `ls`; do cat $i | sed 's/Zarro\ Boogs/No\ Bugs/g'
>$i.tmp; mv $i.tmp $i; done"

>>The current GNATS setup only fits 4, 5 and 8.  How does Bugzilla do?
  I think it does pretty well, all things considered.  Is gcc using
Savannah?  If so, we might have some nice "synergy" going here...

I am not subscribed to the "gcc@gcc.gnu.org" mailing list, so I won't
see replies there.  Reply to mozilla-webtools@mozilla.org or me
individually where I can forward stuff to the core team.

Thanks!


-- 
Matthew P. Barnson
Talk2 Technology, Inc.
"There are three kinds of people in the world.
Those who can count, and those who can't." -unknown

Re: Bugzilla features

[Joseph S. Myers]

On 19 Nov 2001, Matthew P. Barnson wrote:

> 2. Handles MIME in mail.
> 
>   You mean MIME attachments?  Yep, Bugzilla does it, but it doesn't
> currently read MIME type really well.  If it's text, it shows up as
> text, but anything that's binary is simply binary (Microsoft Word .doc
> files and such) if downloaded from the web interface.  Also the binary
> attachment will NOT be forwarded to the mailing list -- only the body
> text will go as a comment.

The main point is that, when someone sends a follow-up by mail, and that 
is QP-encoded (say), it appears cleanly in the bug logs (rather than, as 
at present with GNATS, as raw quoted-printable).  I've explained in my 
other reply
http://gcc.gnu.org/ml/gcc-bugs/2001-11/msg00606.html
that we *do* want attachments to go out in mail.

> 5. Follow-ups sent to gcc-gnats for existing PRs get filed properly in
> any new system.
>   Hmm, mirroring a mailing list to the bugmail system?  That would be
> problematical, but could be done with the right procmail filters in
> place.  It's easier to say that followups to
> "bugzilla-daemon@wherever.org" would go to bugmail, but not replies to
> the mailing list.  Bugzilla's not a mailing list archive system, really.

gcc-gnats is the current address for submissions to GNATS, not a mailing
list - so, for replies for existing bug reports converted into Bugzilla,
we'd need a converter sitting there that sends replies onto
bugzilla-daemon.  New messages sent out about bugs to people and the lists
can include bugzilla-daemon somewhere in their header addresses so that
replies will go there and get properly filed.

> difference with it and gnats... Bugzilla, by default, is not an "open
> bug receiver".  I have a patch to enable automatic bug creation upon
> emailed bug submission, but I can tell you from experience that this is
> an open door for Spammers to get a large target audience.  

Experience shows this not to be a problem with the GNATS setup.  When spam
goes to the gcc-gnats address, it gets filed as a "pending" PR since it
isn't properly formatted as a bug submission, and doesn't get sent to the
lists, so few people see it.

Having a converter that converts current gccbug output (received by
gcc-gnats) into whatever format Bugzilla would expect for emailed bug
submission, and if appropriate a more Bugzilla-tailored new gccbug script
as well, is I think essential.  The detail of whether it then creates
Bugzilla accounts for the people who submit bugs this way I don't care
about.

Spam will not be properly formatted as a bug report from a bug reporting
script.  I don't know the best treatment of such mails that can't be
parsed as bug reports or follow-ups to existing ones, but perhaps they
should go to a moderator who discards them if spam or enters them manually
if relevant.

>   Bugzilla is currently back-ended into MySQL (see
> http://www.mozilla.org/projects/bugzilla/ ).  Dave Lawrence of Redhat
> has done an oracle port, see http://bugzilla.redhat.com/.  Dawn Endico
> made Bugzilla's XML DTD the industry-standard bug-tracking DTD (no
> standardization committee, but most open bug-trackers support the format
> now).  So you can do a mysql dump of data, export it via XML for
> importation elsewhere, or export your existing data to an
> easily-understood XML format for importation.
>   Bugzilla includes a "sanity check" to find basic database issues with
> data formats.  It's not comprehensive, however.  I recommend a good
> daily backup strategy for any Bugzilla database, which is covered in
> great detail at the mysql.com website.

OK, so the system could daily lock the database, do an XML dump and make
that available for rsync so we keep the bulk anonymous access to the whole
database, and back it up as documented?  (The FSF machines have reasonable
backup procedures, but I don't know about sourceware.)  Is the database of
bugs cleanly separated from other information such as user passwords (so
there are no security issues with having it available for anon rsync)?

>   I think it does pretty well, all things considered.  Is gcc using
> Savannah?  If so, we might have some nice "synergy" going here...

GCC isn't - though RMS would I think like it to use the FSF machines more.

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: Bugzilla features

[Matthew P. Barnson]

Apologies for not participating in the gcc-bugs mailing list myself :) 
I'm subscribed to many, many mailing lists.  I also am replying as my
mozilla-webtools normal account holder -- didn't mean to send the
previous mail from my talk2 address.

On Mon, 2001-11-19 at 09:23, Joseph S. Myers wrote:
> The main point is that, when someone sends a follow-up by mail, and that 
> is QP-encoded (say), it appears cleanly in the bug logs (rather than, as 
> at present with GNATS, as raw quoted-printable).  I've explained in my 
> other reply
> http://gcc.gnu.org/ml/gcc-bugs/2001-11/msg00606.html
> that we *do* want attachments to go out in mail.

  OK, then, it would definitely involve a volunteer hacking bug_email.pl
to allow this functionality, and preferably providing the patches back
as a parameter-izable option.  Not in there right now, it only sends a
link to the attachment ID in the database.

> gcc-gnats is the current address for submissions to GNATS, not a mailing
> list - so, for replies for existing bug reports converted into Bugzilla,
> we'd need a converter sitting there that sends replies onto
> bugzilla-daemon.  New messages sent out about bugs to people and the lists
> can include bugzilla-daemon somewhere in their header addresses so that
> replies will go there and get properly filed.

  OK, you can just modify the header via our administration interface in
the "editparams.cgi" script.  There's an option there for the "To:" and
"From:" headers so you can chop them up how you'd like -- change "From:"
to "gcc-gnats@gnu.org", then set up your Bugzilla procmail filters on
that alias.  Trivial, really.
  There's this little part of me that cringes at seeing Bugzilla being
called gnats, but that's just me :)

> Experience shows this not to be a problem with the GNATS setup.  When spam
> goes to the gcc-gnats address, it gets filed as a "pending" PR since it
> isn't properly formatted as a bug submission, and doesn't get sent to the
> lists, so few people see it.
  Understood.  You'd definitely have some kind of converter front-end to
bug_email.pl which converts it into a format bug_email.pl understands
(or, alternately, modify bug_email.pl to accept gccbugs format).  You
configure bug_email.pl to stick submissions which don't specify any
defaults into some working queue.  It works the same.  As long as you
have someone who doesn't mind sifting through the spam in that queue
from time to time : )

> Having a converter that converts current gccbug output (received by
> gcc-gnats) into whatever format Bugzilla would expect for emailed bug
> submission, and if appropriate a more Bugzilla-tailored new gccbug script
> as well, is I think essential.  The detail of whether it then creates
> Bugzilla accounts for the people who submit bugs this way I don't care
> about.

  Right, it's as irrelevant to you as it was to me when I used this
system.  Another option is to strip out sender headers entirely in
bug_email.pl, and make it so that anything coming into the script is
sent in as a certain currently existing user.  Six of one, half a dozen
of the other.
  An important part, too, is setting up procmail filters to filter out
bounces.  Otherwise you can get nasty mailstorms from Bugzilla
auto-replying to an auto-reply address.  That causes database issues,
too.

> Spam will not be properly formatted as a bug report from a bug reporting
> script.  I don't know the best treatment of such mails that can't be
> parsed as bug reports or follow-ups to existing ones, but perhaps they
> should go to a moderator who discards them if spam or enters them manually
> if relevant.

  Yep, that's a problem that has a wetware solution :)

> OK, so the system could daily lock the database, do an XML dump and make
> that available for rsync so we keep the bulk anonymous access to the whole
> database, and back it up as documented?  (The FSF machines have reasonable
> backup procedures, but I don't know about sourceware.)  Is the database of
> bugs cleanly separated from other information such as user passwords (so
> there are no security issues with having it available for anon rsync)?

  Yes, that's almost exactly what I did the last place I adminned
Bugzilla.  The user information table is separate from the bugs table,
but you will get some nasty errors if a bug doesn't have matching user
information when you run a query.  Passwords are encrypted in the
database, but any diligent hacker can eventually pry passwords out of a
hash, particularly if they have offline access.
  Exporting all the bugs in the database as XML, while
processor-intensive, would allow decent replicatability without the
above issue.  Or something which exports everything via mysqldump, then
replaces all the password information with some hash, and also doesn't
export the session table.  However you do it, it would have to be done
with care that you don't expose the sessions and user data.
  So, umm, I guess the answer is "yes and no".  There are security
issues with simply doing a raw dump of the database on a nightly basis
and allowing anon access to it.  They can be worked around with some
diligence on the part of the backup maintainer.

HTH!

-- 
Matthew P. Barnson
Bugzilla Documentation Guru

Re: Bugzilla features

[Joseph S. Myers]

On 19 Nov 2001, Matthew P. Barnson wrote:

>   OK, you can just modify the header via our administration interface in
> the "editparams.cgi" script.  There's an option there for the "To:" and
> "From:" headers so you can chop them up how you'd like -- change "From:"
> to "gcc-gnats@gnu.org", then set up your Bugzilla procmail filters on
> that alias.  Trivial, really.
>   There's this little part of me that cringes at seeing Bugzilla being
> called gnats, but that's just me :)

Bugzilla doesn't need to call itself GNATS here - we could more cleanly
have gcc-gnats as the compatibility address that handles converting
messages that are follow-ups to GNATS PRs or new submissions via the
existing gccbug script, and have gcc-bugzilla (maybe multiple such
addresses for new submissions, follow-ups, control messages, ...,
depending on how it works) be the address appearing in messages Bugzilla
generates.  (In principle the correct thing to do is for messages to have
in their From headers both the Bugzilla address and the address of the
person who made the change / PR submission that generated the message -
perfectly valid according to RFC822/2822, but I don't know if it would
cause problems with any mail clients.)

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: Bugzilla features

[Gerald Pfeifer]

On Mon, 19 Nov 2001, Joseph S. Myers wrote:
> (In principle the correct thing to do is for messages to have
> in their From headers both the Bugzilla address and the address of the
> person who made the change / PR submission that generated the message -
> perfectly valid according to RFC822/2822, but I don't know if it would
> cause problems with any mail clients.)

Not with any sensible client I am aware of (I know because I used such
a scheme now and then).

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/