Re: The TODO list and planning a release schedule

[Mel Hatzis <hatzis@juniper.net>]

Hi Chad, thanks for putting this together...it's a great starting point
for where we're going next. I have more comments below, but before we
delve into them, let me first say thanks for stepping up and taking on
the role of GNATS maintainer.

On 06/15/2004 12:28 PM, Chad Walstrom submitted:
> I figured I'd throw this out for discussion to see what people feel are
> priorities for the GNATS software and what type of release schedule we
> should try to hold to.  The following list is the text output of the XML
> .todo file that has been checked into CVS.
> 
> I'm fishing for comments here, so please let us know what your
> impressions are.  I've copied in almost all of the old TODO list into
> the 'Unassigned' list item.  These are trivial to reparent to different
> releases.  If a task isn't "assigned" to someone, it's because this is
> still a work in progress or because no one has mentioned interest in it.
> 
> Also, our GNATS database, although recovered from the old fencepost
> backups, has not been reinstalled or reinstated.  Consider every release
> a bugfix release, although notable ones may show up on this list.  I'm
> awaiting a fencepost account, and we have requested that the GNATS
> database be installed and restored to its former glory from the GNU
> sysadmin's.  Adam or I will keep you all posted.

Cool.

> 
> A '-' in front of a list item indicates it's finished.
> 
> As far as a timeline, I would like to start preparing a 4.1 pre-release
> ASAP.  There have been some CVS updates since the original 4.0 release,
> and it's a good time mark the codebase before branching off for the big
> changes in 4.2 and 5.0.
> 
>     GNATS General TODO List
>   1.Release 4.1: Code cleanups, bug fixes, documentation updates
>     Added: Fri Jun 11 10:36:54 2004  Completed: incomplete
>     Duration: incomplete  Priority: high
> 
>     - 1.Prepend $(DESTDIR) to Makefile.in installation targets (Chad)
>         Added: Fri Jun 11 11:08:38 2004  Completed: completed on Fri Jun 11 14:07:23 2004
>         Duration: 2h 58m  elapsed  Priority: veryhigh

While we're here, if you try building from a directory other than
the top-level source directory, say by creating $topdir/objdir
and running ../configure from the objdir directory, the --with-submitter
and --with-organization configure flags end up getting ignored.

I've got a fix for this...which basically involves removing all
references to the DEFAULT_ORGANIZATION and SUBMITTER macros from the
$topdir/gnats build files - thereby preventing the config-send-pr
target from screwing things up for send-pr.

I've also got a patch for the doc/Makefile.in so that no attempt
is made to build the info targets if the 'makeinfo' program cannot
be found.

Finally, it'd be really cool if we could add 'make test' to the
build/install process. The real difficulty in achieving this, IMO,
is that there's no way to run gnatsd without first installing it
...since it looks for the databases file. Perhaps adding a command
line switch to gnatsd for specifying the path to the databases
file would work around this limitation.

I've put together a regression test suite which I may be able to
work into the build process. Unfortunately, it uses a small perl
component for setting up the database which will need to be
reworked as a shell script.

> 
>         (Committed this to CVS two days ago.)
>       2.Add missing manpages -- one manpage for each application
>         Added: Fri Jun 11 11:11:57 2004  Completed: incomplete
>         Duration: incomplete  Priority: high
> 
>       3.Kill install-sid script and update documentation
>         Added: Fri Jun 11 11:09:17 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>       4.Update Documentation: Set developer's policy.
>         Added: Fri Jun 11 14:45:45 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

All the above are good. I need to dig around...I have many bug
fixes to provide. Since they've been made in the context of the
gnats-dbi project I've been working on, it's going to be a little
bit of a hassle finding them all and putting patches together.

> 
>   2.Release 4.2: New Features
>     Added: Fri Jun 11 10:37:02 2004  Completed: incomplete
>     Duration: incomplete  Priority: medium
> 
>       1.Mail handling enhancements
>         Added: Fri Jun 11 14:48:24 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>           1.Mail-based manipulation of GNATS database (Mel)
>             Added: Fri Jun 11 14:49:11 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium
> 
>           2.Trigger-based mail format replies (Mel)
>             Added: Fri Jun 11 14:49:28 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium
> 
>           3.Enable To: address@bogus.example.com syntax to queue-pr
>             Added: Fri Jun 11 14:49:40 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium

I'm not clear on #3...can you elaborate.

Also, I've recently implemented something else you might find
valuable...it's a filter which you can apply to a mail-format
for limiting the list of recipients. This is useful in the
context of a view into the PR database...which is possible
with the Oracle backend.

e.g. You might allow a certain class of user to create PRs
but wish to prevent that class of user from receiving particular
GNATS mail notifications.

The feature is also useful in preventing mail loops.

The dbconfig syntax (which is entirely optional) is as follows:

mail-filter "foo" {
     # don't allow email to fred
     deny "^fred.*"
     # allow foo internal email addresses
     allow "[[:alnum:]._-]\+(@foo\.com)\?$"
     # all other email addresses are disallowed
     policy deny
}

mail-format "initial-response-to-submitter" {
     trigger on-create "Category!=\"sw-tools\""
     ...other mail-format elements
     apply-filter "foo"
     ...other mail-format elements
}

The above example would restrict the initial-response-to-submitter
notification so that it would only go to users with a foo email
address (or a non-qualified/internal address).

> 
>       2.MIME Handling
>         Added: Fri Jun 11 14:50:49 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>           1.Detach files appropriately for GNATS DB format
>             Added: Fri Jun 11 14:51:48 2004  Completed: incomplete
>             Duration: incomplete  Priority: high
> 
>           2.Convert HTML to TXT?
>             Added: Fri Jun 11 14:51:15 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium

I like these...I started scoping this using the MIME::Parser and
MIME::Decoder perl modules but didn't get very far. MIME handling
could also support an attachment feature which I would love to
see added to the project.

> 
>   3.Release 5.0: New Features, Major Changes to DB Layer, RDBMS
>     Added: Fri Jun 11 10:37:07 2004  Completed: incomplete
>     Duration: incomplete  Priority: medium
> 
>       1.Database Enhancements
>         Added: Fri Jun 11 14:53:57 2004  Completed: incomplete
>         Duration: incomplete  Priority: high
> 
>           1.DB Abstraction Layer (Mel)
>             Added: Fri Jun 11 14:52:27 2004  Completed: incomplete
>             Duration: incomplete  Priority: high
> 
>           2.Oracle RDBMS Backend
>             Added: Fri Jun 11 14:52:39 2004  Completed: incomplete
>             Duration: incomplete  Priority: high

You might want to add PostgreSQL as a 3rd item. I've started putting
this together.

The development for the Oracle backend is pretty much done...we've been
using it at Juniper for about a year and no additional effort has
been required. This will largely be a matter of review and integration.

A 4th item for Release 5.0 features might be support for transactions.
After doing away with the need for a global gnats lock (using the
Oracle backend), it has become an interesting challenge to see how
far one can go in removing the need for a user to take out a PR lock.
In organizations with large user bases and large numbers of PRs,
the current method of having an edit client lock the PR is sub-optimal.

> 
>       2.Mail handling enhancements
>         Added: Fri Jun 11 14:55:10 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>           1.Maintain mbox archive of all emails
>             Added: Fri Jun 11 14:55:27 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium
> 
>           2.Fake Audit trail entries as emails and append to mbox archive
>             Added: Fri Jun 11 14:55:41 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium
> 
>           3.Continue to use existing PR datafile for logging events, keywords, and metadata
>             Added: Fri Jun 11 14:55:58 2004  Completed: incomplete
>             Duration: incomplete  Priority: medium
> 
>       3.Account enhancements
>         Added: Fri Jun 11 14:56:29 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>           1.PAM Authentication (Pankaj)
>             Added: Fri Jun 11 14:56:45 2004  Completed: incomplete
>             Duration: incomplete  Priority: high
> 
>             (Pankaj expressed interest in this on 2004-06-14)
>   4.Unassigned
>     Added: Fri Jun 11 13:50:51 2004  Completed: incomplete
>     Duration: incomplete  Priority: medium
> 
>       1.Add squirrels, to make pst happy.
>         Added: Fri Jun 11 14:42:32 2004  Completed: incomplete
>         Duration: incomplete  Priority: high
> 
>       2.Add conditional formats
>         Added: Fri Jun 11 13:56:21 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Need more information - what is this?

> 
>       3.Add script hooks, probably for edit formats and such.  Need to think about how to
>         integrate in changes to the PR done by the script
>         Added: Fri Jun 11 13:57:38 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Ahh, script hooks. This would be extremely valuable.

> 
>         (after all, that's really the whole point)
>       4.Allow fields to not exist; add a "field-always-present" option to the field description
>         Added: Fri Jun 11 13:58:17 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 

Need more information - what is this?

While we're here, together with a work colleague, we've revised the way
'required' fields are implemented. Our primary motivation was to simplify
the way they're expressed in the dbconfig and also to enhance the validation
logic (check_pr) so that missing required field errors are caught prior to a
PR being submitted.

This feature does away with the requirement to have an "on-change { require }"
clause in each required field along with a "require { field }" clause in the
initial-fields section of the dbconfig. These are replaced with a simple
"required" flag in the field definition.

We're adding an optional condition to this flag...so that it's possible
to have conditional required fields. The value of doing it this way
is that the dbconfig syntax required for defining a required field is
localized to the field definition and not spread out in different
areas of the dbconfig. This makes it very easy to follow the "required
field logic" in the dbconfig and assists the GNATS administrator in
preventing errors.

>       5.Add virtual fields.
>         Added: Fri Jun 11 13:59:28 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>         (The output format can use the existing format mechanism. This would necessitate having a
>         "no-display" field flag, so that some fields don't show up in a "full" query (adding all
>         the fields to full" is a rather obnoxious requirement).  There should also be a "raw"
>         query that dumps all the real fields as raw contents, for editing purposes (and perhaps
>         for other things).)
>       6.Revise access control mechanisms.
>         Added: Fri Jun 11 14:39:12 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Cool...I think this would be a valuable endevour.

> 
>       7.Should PRs have a ">Database-Name:" header?  Probably, and probably immutable.  Can be
>         used when editing a PR, or submitting an initial one.
>         Added: Fri Jun 11 14:39:27 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Hmmm, this reminds me of another feature I've been considering.
It'd be valuable to investigate the use of non-numeric PR ids.
For example, a 3 letter prefix to the PR number might provide
a useful way to route PRs to a specific GNATS database.

> 
>       8.Append-only fields.  Need to revise access control first.
>         Added: Fri Jun 11 14:40:05 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>       9.Document functions.  Many of them are undocumented (even newly-added ones, shame).
>         Added: Fri Jun 11 14:41:10 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>      10.See how much farther we can go with removing knowledge of particular fields from the
>         gnats code.
>         Added: Fri Jun 11 14:42:03 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Yes...I think this would be a valuable endevour.

> 
>      11.Make it possible to include adm field contents in the configuration file, instead of
>         always using an external config file.
>         Added: Fri Jun 11 14:42:59 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

I'm not sure I see the benefit.

> 
>      12.The database state is not clean.  There should be a struct that describes the current
>         database, and is passed around as needed.
>         Added: Fri Jun 11 14:43:22 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

Isn't this the DatabaseInfo struct?

> 
>      13.The client state is not clean.  The API is horrid; clients should not know or care if
>         they're communicating via the network or locally.  The original solution was to just
>         allow network access, but that's not really fixing the problem.  (We'll know we're there
>         when gnatsd can act as a relay.)
>         Added: Fri Jun 11 14:43:55 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium

localhost mode must die. If the original solution was to do away with localhost mode
and rely solely on network mode, why doesn't this fix the problem exactly?

> 
>      14.The client connection to the server should also be encapsulated in a struct as well. That
>         is, something to describe the client (its hostname, username, password, access level). 
>         Could eventually allow for a single server process that handles multiple connections.
>         Added: Fri Jun 11 14:44:32 2004  Completed: incomplete
>         Duration: incomplete  Priority: medium
> 
>      15.Change edit-pr to include the "Changed-Why:" header in the initial PR template instead of
>         a separate prompt.  Maybe.
>         Added: Fri Jun 11 14:24:46 2004  Completed: incomplete
>         Duration: incomplete  Priority: low

Or, at least bounce the user into a second edit session to specify the reason.
The current method of prompting the user for the reason provides no avenue of
escape if the user decides they want to change a line above.

> 
>      16.Should all the fields listed in the input section be required? Configurable? How about
>         rejecting initial erroneous PRs (PRs with bad fields) instead of fixing them up? It sucks
>         that pr-edit --submit < /dev/null could quite presumably create a valid PR.
>         Added: Fri Jun 11 14:25:50 2004  Completed: incomplete
>         Duration: incomplete  Priority: low

I would argue that they should *not* all be required. Configurable yes...this is
essentially what I described above in #4 I believe.

> 
>      17.The initial PR filing stuff is way too complicated.  In particular, the various field
>         checks should be configured in dbconfig.  That would let us remove more builtin fields.
>         Added: Fri Jun 11 14:38:18 2004  Completed: incomplete
>         Duration: incomplete  Priority: low

Yep. Anything we can do to remove the builtin fields is worthwhile.

> 
>      18.Decide if the "exec gnatsd locally" option is a security hole.  (Probably.) Make it #if
>         TESTING only?
>         Added: Fri Jun 11 14:38:30 2004  Completed: incomplete
>         Duration: incomplete  Priority: low
> 
>      19.Come up with a better name for the lexer source file.
>         Added: Fri Jun 11 14:41:35 2004  Completed: incomplete
>         Duration: incomplete  Priority: low
> 

I've grown rather fond of fconfigl.l ;-)

Extending on these unassigned items, it'd be wothwhile considering how one might
go about creating dynamic views - i.e. to narrow down on applicable field values based
on the selection of other field values. As an example, when class is set to 'hw-bug'
only show the h/w categories to the user. This would obviously be handled (or not)
by the client, however, before considering how the client does this, we should
consider how we can capture the necessary information to allow the client to
implement this. I think the dbconfig would be the right place to capture this
and have some ideas about how this might be expressed (which build on the filtering
idea I presented above for mail-format recipients).

Finally, it's also worthwhile considering a PR version feature. Often times, a PR
manifests itself differently on different platforms, or is in different states for
different releases. I've put together a rough draft of a design document for
how this might be done...which I'd be happy to share with you after I review it a
few more times.

--
Mel Hatzis


_______________________________________________
Help-gnats mailing list
Help-gnats@gnu.org
http://lists.gnu.org/mailman/listinfo/help-gnats