proclamation of intent

proclamation of intent

[Bob Manson]

Or something like that, anyway.

I recently started working for Juniper, and was asked to look into
doing some gnats development (specifically, making it easy to add new
fields).  I asked the almighty Jason "Wreck" Molenda about getting cvs
access, and he urged me to take on a larger maintainership role.

So I shall.

This has turned into a major hacking frenzy.  I've been rewriting
large bits of code, stripping out hordes of vile global variables,
adding bits of new functionality, and just plain trying to clean it
up.  It's an ugly horrible job, but it badly needs to be done.

There will be breakage. I will try to minimize this, if for no other
reason than that we need to actually use it here.  The external client
interface will be unchanged at least for now, but internally things
are markedly different, and there will be a lot of new options.

The plan is to have all information that describe fields live in a
struct (currently it's all done as extensions of PR_entry, tho this
will probably change), rather than hard-coded throughout the sources.
This work is mostly complete.  Some fields have special meaning to
gnats, such as Number, Category, Submitter, State, Last-Modified,
etc. and thus their presence will remain hardcoded, but wherever
possible fields will be user-configurable, and it will certainly be
possible to add new fields via a configuration file.

The clients will need to obtain the list of valid fields and their
datatypes by doing appropriate queries.  Perhaps it would make sense
to add these sorts of functions to query-pr, tho it might be a better
idea to have a separate program to do it instead.  Such a client would
also be able to do basic field validation (verify that the provided
data is valid for an enumerated field).

The interface to the random administrative files has been made
generic; in essence any field can have an administrative file
associated with it now.  One thought is to move the data contained in
these files into a central file with a better structure (perhaps with
tagged fields etc).

I'm beginning to wonder if it makes sense for the user to invoke
query-pr directly.  That is, it may make sense to move some amount of
the current functionality that is in query-pr into another program,
and have query-pr be a script or some other program that does
appropriate formatting of output.  I've never liked the way fields
were delimited in query-pr's output, and perhaps this would be an
opportunity to change it.

As a specific example, the various query classes that are currently in
query-pr don't exactly make sense to me in a more generic context.
That functionality can be easily achieved by putting an appropriate
script on top of a query-pr-like program.

Obviously there will need to be a configuration file that describes
the field structure.  That's essentially the main issue I need to
resolve now, by coming up with an appropriate design. I'm thinking
something along the lines of:

	field "Release-Note" {
		multitext
		query-default exact-regexp
	}

	field "Release" {
		string matching "insert-regex-here"
		query-pr-option "A" "--release"
		network-query-command "RLSE"
		query-default inexact-regexp
	}

	field "Priority" {
		enum {
			values {
				"high" "medium" "low"
			}
			default "medium"
		}
		query-pr-option "p" "--priority"
		network-query-command "PRIO"
		query-default exact-regexp
	}

	field "Class" {
		enumerated_in_file {
			path "path/relative/to/GNATS_ROOT"
			fields {
				"class" "type" "description"
			}
			key "class"
		}
		query-pr-option "L" "--class"
		network-query-command "CLSS"
		query-default inexact-regexp
	}
	...
	index {
		path "path/relative/to/GNATS_ROOT"
		fields {
			"Submitter"  "Responsible" "State" "Confidential"
			"Severity" "Priority" "Date-Required" "Quarter"
			...
		}
		separator "|"
	}

For now, I will also have a query section that will look something like

	query "Full" {
		fields all
	}

	query "Summary" {
		printf ("%8s %-8.8s %-8.8s %-9.9s %-9.9s %-8.8s %-10.10s %s",
			"Number", "Responsible", "Category", "State",
			"Severity", "Priority", "Submitter", "Synopsis");
	}

	...

This is probably all "a bit verbose", but I like files where I can
just look at them and have a good idea of exactly what's going on.

At some point it would be nice to add "triggers", or the ability to
invoke a program or script to do field verification etc.

The new options will come in when new fields are used.  Obviously the
current scheme of single-letter options or specific network commands
for queries is not extensible or practical; instead, query-pr will have
options such as

	--field-Content --equals "foo"
	--field-Last-Modified --is-less-than "some-date"

I haven't decided the exact syntax of this, but that's the basic idea.
I'd like to be able to specify inexact matches versus exact matches,
and strings versus regexps too.  Maybe I should give up on the idea of
trying to do this as a set of options, and make it full expressions.

There will be appropriate network commands added for querying specific
fields.  This part is relatively obvious.

I've already added the ability to have boolean expressions to query-pr:

( --release "3.3" --and --submitter "bar" ) --or --release "3.4"

along with appropriate extensions to the network protocol.

These changes will necessarily make the network clients incompatible
with old servers, and vice-versa.  I don't see much reason to try and
make this work.

Thoughts, comments, questions, flames?
					Bob

Re: proclamation of intent

[Rick Macdonald]

On Wed, 18 Aug 1999, Bob Manson wrote:

Great news!

> The plan is to have all information that describe fields live in a
> struct (currently it's all done as extensions of PR_entry, tho this
> will probably change), rather than hard-coded throughout the sources.
> This work is mostly complete.  Some fields have special meaning to
> gnats, such as Number, Category, Submitter, State, Last-Modified,
> etc. and thus their presence will remain hardcoded, but wherever
> possible fields will be user-configurable, and it will certainly be
> possible to add new fields via a configuration file.

Sounds good.

> The clients will need to obtain the list of valid fields and their
> datatypes by doing appropriate queries.  Perhaps it would make sense
> to add these sorts of functions to query-pr, tho it might be a better
> idea to have a separate program to do it instead.  Such a client would
> also be able to do basic field validation (verify that the provided
> data is valid for an enumerated field).

At this point I can't tell which would be better yet.

> I'm beginning to wonder if it makes sense for the user to invoke
> query-pr directly.  That is, it may make sense to move some amount of
> the current functionality that is in query-pr into another program,
> and have query-pr be a script or some other program that does
> appropriate formatting of output.

As above, I can't tell yet.

> As a specific example, the various query classes that are currently in
> query-pr don't exactly make sense to me in a more generic context.
> That functionality can be easily achieved by putting an appropriate
> script on top of a query-pr-like program.

By "query classes" do you mean "full", "sql", etc?

> Obviously there will need to be a configuration file that describes
> the field structure.  That's essentially the main issue I need to
> resolve now, by coming up with an appropriate design. I'm thinking
> something along the lines of:
> 

> 	field "Priority" {
> 		enum {
> 			values {
> 				"high" "medium" "low"
> 			}
> 			default "medium"
> 		}
> 		query-pr-option "p" "--priority"
> 		network-query-command "PRIO"
> 		query-default exact-regexp
> 	}

This seems OK.

> 	index {
> 		path "path/relative/to/GNATS_ROOT"
> 		fields {
> 			"Submitter"  "Responsible" "State" "Confidential"
> 			"Severity" "Priority" "Date-Required" "Quarter"
> 			...
> 		}
> 		separator "|"
> 	}

GNATS would then dynamically decide if any given query required only index
fields, or if the entire PR text had to be read? It does that now, but
it's hardwired to know that all but multitext fields are in the index.

> For now, I will also have a query section that will look something like
> 
> 	query "Full" {
> 		fields all
> 	}
> 
> 	query "Summary" {
> 		printf ("%8s %-8.8s %-8.8s %-9.9s %-9.9s %-8.8s %-10.10s %s",
> 			"Number", "Responsible", "Category", "State",
> 			"Severity", "Priority", "Submitter", "Synopsis");
> 	}

This seems inconsistant. Would this be better:

        query "Summary" {
                format "%8s %-8.8s %-8.8s %-9.9s %-9.9s %-8.8s %-10.10s %s"
                fields {
                       "Number" "Responsible" "Category" "State"
                       "Severity" "Priority" "Submitter" "Synopsis"
                {
        }

This opens an interesting possibility. Couldn't any client then feed
something similar to the above to [n]query-pr/gnatsd, and thereby define
it's own query output?

> This is probably all "a bit verbose", but I like files where I can
> just look at them and have a good idea of exactly what's going on.

Of great interest to front-end writers is the format that query-pr and
gnatsd would use to send the above info to the client. Obviously, where
the above examples refer to external admin files, GNATS would send the
expanded field info, and _not_ a reference to any of these files. 

As a start, this format would want to be reasonable to parse as a list in
Tcl and perl. Perhaps we need client-selectable formats: tcl, perl, sh,
csh?

> The new options will come in when new fields are used.  Obviously the
> current scheme of single-letter options or specific network commands
> for queries is not extensible or practical; instead, query-pr will have
> options such as
> 
> 	--field-Content --equals "foo"
> 	--field-Last-Modified --is-less-than "some-date"
> 
> I haven't decided the exact syntax of this, but that's the basic idea.

Seems basically OK.

> I'd like to be able to specify inexact matches versus exact matches,
> and strings versus regexps too.  Maybe I should give up on the idea of
> trying to do this as a set of options, and make it full expressions.

It sounds like you might just use regexps. An exact match of "RickM" would
need ^RickM$. An inexact (substring?) might be .*ick.*, *ick*, or just
ick?

On the other hand, I think I know what you're after. Tcl, for example, has
a list search command (lsearch) with mode switches of -exact, -glob and
-regexp.

> I've already added the ability to have boolean expressions to query-pr:
> 
> ( --release "3.3" --and --submitter "bar" ) --or --release "3.4"

Cool!

> These changes will necessarily make the network clients incompatible
> with old servers, and vice-versa.  I don't see much reason to try and
> make this work.

Agreed.

...RickM...

Re: proclamation of intent

[Bob Manson]

[resent to list, I missed a cc:]

In message < Pine.GSO.4.10.9908181250580.26990-100000@sys4 >, Rick Macdonald writ
es:
>By "query classes" do you mean "full", "sql", etc?

Exactly.

>This seems inconsistant. Would this be better:
>
>        query "Summary" {
>                format "%8s %-8.8s %-8.8s %-9.9s %-9.9s %-8.8s %-10.10s %s"
>                fields {
>                       "Number" "Responsible" "Category" "State"
>                       "Severity" "Priority" "Submitter" "Synopsis"
>                {
>        }

Yeah, much better.  Thanks.  I just pulled the printf stuff off the
top of my head.

>This opens an interesting possibility. Couldn't any client then feed
>something similar to the above to [n]query-pr/gnatsd, and thereby define
>it's own query output?

That's the plan.  Either something like the above formatting, or a
simple list of fields to return as output.

>As a start, this format would want to be reasonable to parse as a list in
>Tcl and perl. Perhaps we need client-selectable formats: tcl, perl, sh,
>csh?

I could see that.  Want to make this as simple as possible to use, so
having that sort of flexibility would be important.

I'm also not sure what exactly the clients would need to know, other
than the names of the fields, their datatypes, and probably
information about any restrictions on what values are valid (for enums
the set of enumerated values, regexps, etc.)  Any further thoughts
would be most helpful.

Oh, and there should be a human-readable description of the field.
Don't want to forget that!

Ideally it would be possible for the client to get all of the data
that describes a field, but presenting this in a useful way is going
to be difficult (this quickly turnes into the traditional textual
database metadata problem).

What I want to avoid is getting *too* generic.  Somewhere there's a
rather thin line between "this is a problem reporting/bug tracking
system" and "this is a generic searchable text database that happens
to have some default settings oriented around bug tracking".  I think
that happens about the time you have to try and figure out how to
describe fields other than in very simplistic terms.

The *eventual* goal is to be able to layer this on top of a "real"
database.  But that's off in the future.

>It sounds like you might just use regexps. An exact match of "RickM" would
>need ^RickM$. An inexact (substring?) might be .*ick.*, *ick*, or just
>ick?

Well, sometimes you really want to specify a string, even if it has
regexp characters in it.  Obviously you can escape everything, but
that can be painful (especially for an unknowing user that's calling a
simple client like query-pr, and passing in abbreviations to search
for).

I can also envision different search types for other data types,
primarily numeric fields.  It'd be sorta nice to be able to select
ranges of PRs easily (without having to explain complex regexps to
users), or other types of numeric fields (numeric priorities?  version
numbers or release qualifiers that are actual numbers?)  If you have
that ability, then being able to specify string versus regexp searches
seems like a natural extension.

As another example, perhaps we have several severity levels, and we
want all PRs that are less than serious.  Yes, we can list all of the
less-than-serious priorities out in a regexp search, but it's far
easier and more accurate to specify that we want all PRs less than
serious.

As for the string versus substring match, query-pr currently has
varying behavior depending on what fields are being searched (some
fields trying to match with an exact regexp, others allow matches with
parts of the field).  I'm not sure if this is good or bad (the choices
really seem quite arbitrary) but that's the way it's worked for a long
time.  I'd rather give the user the ability to specify it, perhaps
with defaults specified in the configuration file.

As for when this will be released... I was hoping to get something out
this week, probably without any of the configuration file code but
with a lot of the other changes I've been proposing.
					Bob

Re: proclamation of intent

[Stephen Dahmen]

Hi Bob...

great ideas...

not sure if this is related to gnats, or just gnatsweb (which is the only
way I interface with gnats at the moment)... but it would be nice to
specify a sort query results.

Any thoughts about rewriting it using Perl instead?  here's some bennys
that I see:

1. you could use DBI and start with flat file database(s), then move to a
"real database" by just changing DBD modules. In fact users could decide to
use whatever database they want. Then you'll grant them SQL capability and
you will have many friends and future employers.

2. Take you half the time of doing it in C. 

3. Would have a fresh design rather than a severely munged C implementation.

4. It could be object oriented rather than arrgh structured.

5. Really portable! Even to NT probably! 

6. Perl already has an awesome distribution mechanism in place (CPAN)

7. would probably multiply the number of teams using it.

8. Would make it easier for a web interface to be written (could hook/reuse
	the core code)

9. It could much more easily be extended by future GNATS junkies.


Course if you don't know Perl, then it could be a drawback... :\
I'd be willing to help out informally with the design/architecture if you'd
like.

Good Luck with it.  Lets hope the client developers take advantage of it...
my thought is fewer and fewer are going to be using GNATS from the command
line.. i know my team couldn't use it that way.  Gnatsweb has been a
blessing. I'd hope you would start to lean more towards facilitating web
integration .

Oh, and there should be a way to convert "old" gnats data into the new
world without having to re-enter everything again.  Another good job for Perl.

Regards,

Stephen

PS> is there any affiliation between Juniper and the current keepers of
GNATS that would afford you the luxury of doing all this work? Just curious!

>

Re: proclamation of intent

[Rick Macdonald]

On Wed, 18 Aug 1999, Bob Manson wrote:

> >As a start, this format would want to be reasonable to parse as a list in
> >Tcl and perl. Perhaps we need client-selectable formats: tcl, perl, sh,
> >csh?
> 
> I could see that.  Want to make this as simple as possible to use, so
> having that sort of flexibility would be important.
> 
> I'm also not sure what exactly the clients would need to know, other
> than the names of the fields, their datatypes, and probably
> information about any restrictions on what values are valid (for enums
> the set of enumerated values, regexps, etc.)  Any further thoughts
> would be most helpful.
> 
> Oh, and there should be a human-readable description of the field.
> Don't want to forget that!

Yes, absolutely!

I feel that most of the "client hint" stuff that I sent out a while ago
still applies. IE which fields are to be shown in send PR, which trigger
email and change reasons and audit-trail entries, etc.

Have you seen all that? If not, I've pasted it at the bottom here.

I think these items could just be additional fields in your proposal. eg:

        field "Release-Note" {
                multitext
                query-default exact-regexp
		send-suppress
		edit-requires-reason
		edit-requires-audit-trail
		mandatory
        }

Or it could be like gnatsweb:

                flags "sendsuppress|editreason|audittrail|mandatory"

Or whatever; I'm easy.

> Ideally it would be possible for the client to get all of the data
> that describes a field, but presenting this in a useful way is going
> to be difficult (this quickly turnes into the traditional textual
> database metadata problem).

I don't see anything that the client _wounldn't_ want to get...

> >It sounds like you might just use regexps. An exact match of "RickM" would
> >need ^RickM$. An inexact (substring?) might be .*ick.*, *ick*, or just
> >ick?
> 
> Well, sometimes you really want to specify a string, even if it has
> regexp characters in it.  Obviously you can escape everything, but
> that can be painful (especially for an unknowing user that's calling a
> simple client like query-pr, and passing in abbreviations to search
> for).
> 
> I can also envision different search types for other data types,
> primarily numeric fields.  It'd be sorta nice to be able to select
> ranges of PRs easily (without having to explain complex regexps to
> users), or other types of numeric fields (numeric priorities?  version
> numbers or release qualifiers that are actual numbers?)  If you have
> that ability, then being able to specify string versus regexp searches
> seems like a natural extension.
> 
> As another example, perhaps we have several severity levels, and we
> want all PRs that are less than serious.  Yes, we can list all of the
> less-than-serious priorities out in a regexp search, but it's far
> easier and more accurate to specify that we want all PRs less than
> serious.

All of these cool things would be great.

> As for the string versus substring match, query-pr currently has
> varying behavior depending on what fields are being searched (some
> fields trying to match with an exact regexp, others allow matches with
> parts of the field).  I'm not sure if this is good or bad (the choices
> really seem quite arbitrary) but that's the way it's worked for a long
> time.  I'd rather give the user the ability to specify it, perhaps
> with defaults specified in the configuration file.

The current behaviour sucks; I can never remember when I need .*foo or
just foo!

> As for when this will be released... I was hoping to get something out
> this week, probably without any of the configuration file code but
> with a lot of the other changes I've been proposing.

Did you pick up my file-pr changes? gnatsweb is actually broken without
them.

BTW - I have no idea how many of the regular contributors subscribe to
this list. I hope they are all seeing this.

Here's the client hint stuff. I think just the functionality of
SUPPRESSED_FIELDS and FIELD_ALIASES would be obsolete with your great
additions.

The following are called "client hints" because front-end GNATS
programs such as gnatsweb and TkGnats are free to support any and all
of these configuration options. These options help give GNATS some
degree of user-customizable fields until such time as the core GNATS
code is modified for complete support.

In the absence of the client hints, the default behaviour of any
particular front-end client is up to the discretion of the client.

The format is comma-separated field names. For example:

	MANDATORY_FIELDS="Category,Originator,Synopsis,Description"

These optional parameters may be placed in the GNATS configuration
file gnats-db/gnatas-adm/config. GNATS itself ignores these options.

Where defaults are suggested, they are just a guideline to get the
same behaviour as the gnats clients send-pr and edit-pr.

Please note that GNATS currently has a limit of 255 characters per
line in the config file, including the parameter name.

FIELD_ALIASES:
        You can effectively rename any GNATS field with these aliases.
        The alias only affects the client interface. Behind the 
        scenes, the GNATS PR field still has the original name.

REQUIRE_AUDIT_TRAIL_ENTRY: 
        Edits to these fields trigger entries in the Audit-Trail.
        "none" signals no fields. "all" signals all fields.

REQUIRE_AUDIT_TRAIL_REASON: 
        Edits to one or more of these fields triggers the
        prompt of one reason for all changes for the Audit-Trail.
        "none" signals no fields. "all" signals all fields.

REQUIRE_AUDIT_TRAIL_EMAIL: 
        Edits to one or more of these fields trigger the 
        sending of email notification.
        "none" signals no fields. "all" signals all fields.

MANDATORY_FIELDS:
        SendPR will insist that new PRs have values for these fields.
        EditPR will insist that editied PRs have values for these fields.
        "none" signals no fields. "all" signals all fields, but doesn't
        include Audit-Trail, Unformatted or Release-Note in EditPR
	because these fields are usually added by GNATS and are often
        blank for the first part of a PRs life. You can always add these
        fields if you wish.

SUPPRESSED_FIELDS: 
        Fields to suppress from view in all front-end views.
        SUPPRESSED_FIELDS override MANDATORY_FIELDS if a field
        appears in both lists.

SUPPRESSED_SEND_FIELDS:  See "unless" options below.
        Fields not to present in SendPR.
        Format: field[(unless)]
	Default: State,Responsible,Release-Note,Unformatted,Audit-Trail
        Example: State,Responsible(edit),Release-Note,Audit-Trail

SUPPRESSED_EDIT_FIELDS:  See "unless" options below.
        Fields not to present, or to present read-only, in EditPR.
	It is suggested that front-ends do not allow editing Arrival-Date,
        Last-Modified and Closed-Date, since these are maintained by
GNATS.
        Format: field[(unless)]
        Example: Audit-Trail(admin)

SUPPRESS "unless" options:
        Finer control over the showing of fields in SendPR and EditPR.
        In SendPR, the field is suppressed. 
        In EditPR, the field is suppressed or shown read-only, at the
                   discression or configuration of the front-end.

        Suppress fields:

        user      - unless the user of the front-end asks for it.
        edit      - unless the user has edit access.
        edituser  - unless the user has edit access and
                    the user of the the front-end asks for it.   
        admin     - unless the user has admin access.
        adminuser - unless the user has admin access and
                    the user of the the front-end asks for it.   

...RickM...

Re: proclamation of intent

[Rick Macdonald]

On Wed, 18 Aug 1999, Bob Manson wrote:

> As for when this will be released... I was hoping to get something out
> this week, probably without any of the configuration file code but
> with a lot of the other changes I've been proposing.

What does everybody think about numbering the next release 3.201, or
3.300 or 4.0?

I wish I had a beer for every time I've answered mail from people trying
to build 3.2!

...RickM...

Re: proclamation of intent

[Paul Traina]

If we're backwards compatible with 3.x's API's and GNATS socket interface,
I'm cool with calling it 3.300.  However I note that we have a unique
opportunity to clean up some of the additional configuration mess that
has occured over the years (e.g. gnatsd config files are a nightmare) and
if we want to make backwards incompatible changes, we should do a 4.0 beta
release.

Paul

On Mon, Aug 23, 1999 at 11:18:12AM -0600, Rick Macdonald wrote:
> On Wed, 18 Aug 1999, Bob Manson wrote:
> 
> > As for when this will be released... I was hoping to get something out
> > this week, probably without any of the configuration file code but
> > with a lot of the other changes I've been proposing.
> 
> What does everybody think about numbering the next release 3.201, or
> 3.300 or 4.0?
> 
> I wish I had a beer for every time I've answered mail from people trying
> to build 3.2!
> 
> ...RickM...
> 

Re: proclamation of intent

[Rick Macdonald]

On Mon, 23 Aug 1999, Paul Traina wrote:

> If we're backwards compatible with 3.x's API's and GNATS socket interface,
> I'm cool with calling it 3.300.  However I note that we have a unique
> opportunity to clean up some of the additional configuration mess that
> has occured over the years (e.g. gnatsd config files are a nightmare) and
> if we want to make backwards incompatible changes, we should do a 4.0 beta
> release.

Sounds good; thanks.

...RickM...

Re: proclamation of intent

[Bob Manson]

In message < Pine.GSO.4.10.9908231115550.20253-100000@sys4 >, Rick Macdonald writ
es:
>On Wed, 18 Aug 1999, Bob Manson wrote:
>
>What does everybody think about numbering the next release 3.201, or
>3.300 or 4.0?

I was shooting for 4.0 myself, since I think it's a big enough change
to merit it.  Maybe "4.0-this-is-guaranteed-to-corrupt-your-PRMS-database"
would be a better name ;-)

It's currently not network-protocol-backwards-compatible.  I've
removed the myriad of "QXXX" commands and replaced them all with 
"QUER queryformatname".  I could put back the old ones, I suppose. (On
the other hand if I don't fix this, that means I can rip out all those
screwy LXXX commands, the field designation commands, et al.)

In other news, I've implemented the configuration file, and the format
looks almost exactly like what we came up with. Works too, tho I
haven't removed the old field initialization stuff yet (it's in a new 
source file scheduled for demolition).

Maybe I should work on getting a preliminary release out instead of
finishing up what I originally started, namely the ability to add
fields ;-) I have yet to add the network protocol for dumping out the
field info (and the associated client options), there are still a fair
number of places that know how many fields there are, and the mapping
between internal field names and actual fields is not done (it still
essentially derefs an array, bad juju).

If anyone really wants to play with this, I'll put it up.  Check it
in? Something.  I say check it into CVS and let Patty sort it out.
Other opinions?

>I wish I had a beer for every time I've answered mail from people trying
>to build 3.2!

I wish I had a beer for every line of crap code I've deleted.  I'd be
drunk for a couple of months, easy. ;-)
						Bob

Re: proclamation of intent

[Paul Traina]

How about if you drop a branch tag for gnats 3.x in now (in case folks want
to do later 3.x releases, and commit a snapshot into the head as 4.0-alpha).
This will give Rick and Ken a chance to change their front-ends to work with
4.0's commands, and we'll just screw the old network protocol.

On Mon, Aug 23, 1999 at 10:38:14AM -0700, Bob Manson wrote:
> In message < Pine.GSO.4.10.9908231115550.20253-100000@sys4 >, Rick Macdonald writ
> es:
> >On Wed, 18 Aug 1999, Bob Manson wrote:
> >
> >What does everybody think about numbering the next release 3.201, or
> >3.300 or 4.0?
> 
> I was shooting for 4.0 myself, since I think it's a big enough change
> to merit it.  Maybe "4.0-this-is-guaranteed-to-corrupt-your-PRMS-database"
> would be a better name ;-)
> 
> It's currently not network-protocol-backwards-compatible.  I've
> removed the myriad of "QXXX" commands and replaced them all with 
> "QUER queryformatname".  I could put back the old ones, I suppose. (On
> the other hand if I don't fix this, that means I can rip out all those
> screwy LXXX commands, the field designation commands, et al.)
> 
> In other news, I've implemented the configuration file, and the format
> looks almost exactly like what we came up with. Works too, tho I
> haven't removed the old field initialization stuff yet (it's in a new 
> source file scheduled for demolition).
> 
> Maybe I should work on getting a preliminary release out instead of
> finishing up what I originally started, namely the ability to add
> fields ;-) I have yet to add the network protocol for dumping out the
> field info (and the associated client options), there are still a fair
> number of places that know how many fields there are, and the mapping
> between internal field names and actual fields is not done (it still
> essentially derefs an array, bad juju).
> 
> If anyone really wants to play with this, I'll put it up.  Check it
> in? Something.  I say check it into CVS and let Patty sort it out.
> Other opinions?
> 
> >I wish I had a beer for every time I've answered mail from people trying
> >to build 3.2!
> 
> I wish I had a beer for every line of crap code I've deleted.  I'd be
> drunk for a couple of months, easy. ;-)
> 						Bob

Re: proclamation of intent

[Bob Manson]

In message < 19990823104833.A6147@red.juniper.net >, Paul Traina writes:
>How about if you drop a branch tag for gnats 3.x in now (in case folks want
>to do later 3.x releases, and commit a snapshot into the head as 4.0-alpha).

I'll buy that.

I need to finish up some enum cleanup I'm doing (changing the internal
format of enums into a linked list instead of a string), make sure
everything still sorta works, cook up some quick documentation(?)
updates(?? ;-) and I'll drop it in.

I don't even want to think about writing ChangeLog entries for this
stuff ;-)
						Bob

Re: proclamation of intent

[Paul Traina]

Now's a great time to rotate all that crap out anyway.  Start the changelogs
fresh. :-)

On Mon, Aug 23, 1999 at 10:51:45AM -0700, Bob Manson wrote:
> In message < 19990823104833.A6147@red.juniper.net >, Paul Traina writes:
> >How about if you drop a branch tag for gnats 3.x in now (in case folks want
> >to do later 3.x releases, and commit a snapshot into the head as 4.0-alpha).
> 
> I'll buy that.
> 
> I need to finish up some enum cleanup I'm doing (changing the internal
> format of enums into a linked list instead of a string), make sure
> everything still sorta works, cook up some quick documentation(?)
> updates(?? ;-) and I'll drop it in.
> 
> I don't even want to think about writing ChangeLog entries for this
> stuff ;-)
> 						Bob

gnatsd accepting new PRs

[Rick Macdonald]

Bob - back in April, Tony Parent contributed a patch to allow gnatsd to
accept new PRs directly. This is a feature that everyone has wanted for a
long long time. One benefit is that front-ends can notify the user
immediately of the new PR number.

I don't suppose that this ever got picked up? The patch probably wouldn't
match your new code now.

Here are the notes that Tony last sent, just to give you an idea of what
was done. I have the "version 3" of the actual patch, if you want it. I
never installed the patch itself, but I gave a little bit of feedback to
the first couple of attempts to help Tony as he was doing it.

--- snip ---

Here is version 3 of a patch to gnats-3.110 that changes gnatsd so that it
can create new PR's directly. It adds a new command to gnatsd "crte" that
takes a PR text followed by a "." on a line by itself.

The basic difference from version 2 of this patch is that I no longer use
setjmp/longjmp in punt(). The code is still in there, but is turned off
for now via a #ifdef. I left the setjmp/longjmp in just in case. You may
discard this if you don't want it. Not using setjmp/longjmp makes the
gnats() function in file-pr.c more convuluted, but hopefully more
portable.

Procedure:
 - Check for db lock. -- print error and return if locked
 - Lock db if unlocked.
 - Read in text.
 - Call gnats() function from file-pr to create new PR
 - Unlock db.
 - Print out PR number. (Return PR # to client.)

Some things I had to change:

Makefile   -- to add file-pr.o to gnatsd

cmds.c     -- to add the command. Other changes in this file include a
quite
              flag so reset and unlock don't print out stuff if an error
              occured, and the addition of the crte command to the help
              function.

file-pr    -- Changed the function "gnats()" to return an integer. This is
              the new PR number or -1 if none. It used to return void so
              all previous calls will simply discard this value. Therefore
              this should be safe.

gnatsd.c   -- Added the command to the list of possible commands.

...RickM...

Re: gnatsd accepting new PRs

[Bob Manson]

In message < Pine.GSO.4.10.9908231147140.20253-100000@sys4 >, Rick Macdonald writ
es:
>I don't suppose that this ever got picked up? The patch probably wouldn't
>match your new code now.

It's certainly not in the source tree I've been working with, but I
suspect it can be made to work without too much trouble.  It may be
even easier now, I don't know.  (At least it *should* be easy to
implement, so if it's not that needs to be fixed.)

Nothing in the "backend" code calls punt, or I should say "should call
punt" so if that was the fundamental problem that's long since been
addressed.
						Bob

Re: proclamation of intent

[Rick Macdonald]

On Mon, 23 Aug 1999, Paul Traina wrote:

> How about if you drop a branch tag for gnats 3.x in now (in case folks want
> to do later 3.x releases, and commit a snapshot into the head as 4.0-alpha).
> This will give Rick and Ken a chance to change their front-ends to work with
> 4.0's commands, and we'll just screw the old network protocol.

I guess I'll buy that. Just make it the best you can, and we front-end
writers will go with it.

A few minor comments. I think you already have this issues "done right":

The fields and enums should have human-readable Help strings (multi-line
allowed?), and these strings need to be available to the client.

I would like that all configuration stuff is read and parsed by GNATS, and
when sent to the client, it is formatted and written by GNATS (gnatsd)
such that the client just parses it and doesn't have to check for errors
for remove comments. By this I mean that GNATS shouldn't just copy a gnats
admin file to the output (gnats used to do this; I removed a few cases of
it). This also makes the formatting of the info more consistent for
parsing by clients.

I think you should send out your proposed format of configuration info
that will be sent to clients that ask for it. I think TkGnats (me) and
gnatsweb (Ken and Matt) are the only people using this feature. I'd hate
for you to code a bunch of stuff only to have us bug you to change it
afterwards! Tcl and perl are great at parsing, but we might as well make
it simple for ourselves...

...RickM...

Re: gnatsd accepting new PRs

[Tony Parent]

On August 23, 1999, Bob Manson wrote:
> In message < Pine.GSO.4.10.9908231147140.20253-100000@sys4 >, Rick Macdonald writ
> es:
> >I don't suppose that this ever got picked up? The patch probably wouldn't
> >match your new code now.
> 
> It's certainly not in the source tree I've been working with, but I
> suspect it can be made to work without too much trouble.  It may be even
> easier now, I don't know.  (At least it *should* be easy to implement, so
> if it's not that needs to be fixed.)
> 
> Nothing in the "backend" code calls punt, or I should say "should call
> punt" so if that was the fundamental problem that's long since been
> addressed.
> 						Bob

As I remember it, the problem was that the code in file-pr.c called punt
(for things like missing fields etc), but punt calls exit. (So if a
required field was missing, the PR wasn't created.) I had to modify
the punt code so it wouldn't exit the server if file-pr had an error, but
simply return to the main loop of the server.

It's probably good that you haven't seen my code, or you would had more
"crap code" to delete. :-)

-- 
===========================================================================
     oO   Tony.Parent@lsil.com      LSI Logic                         __o
    []<                             2001 Danfield Ct                 -\<,
      |   (970) 206-5769            Fort Collins, CO 80525          O / O
===========================================================================

Re: proclamation of intent

[Tony Parent]

On August 23, 1999, Rick Macdonald wrote:
[...]
> 
> I think you should send out your proposed format of configuration info
> that will be sent to clients that ask for it. I think TkGnats (me) and
> gnatsweb (Ken and Matt) are the only people using this feature. I'd hate
> for you to code a bunch of stuff only to have us bug you to change it
> afterwards! Tcl and perl are great at parsing, but we might as well make
> it simple for ourselves...

I also use it in my Java front end, and Java isn't as good at parsing as
perl/tcl, so I would add my vote to making it (parsing) as easy and
consistent as possible.

-- 
===========================================================================
     oO   Tony.Parent@lsil.com      LSI Logic                         __o
    []<                             2001 Danfield Ct                 -\<,
      |   (970) 206-5769            Fort Collins, CO 80525          O / O
===========================================================================

Re: proclamation of intent

[Rick Macdonald]

On Mon, 23 Aug 1999, Tony Parent wrote:

> I also use it in my Java front end, and Java isn't as good at parsing as
> perl/tcl, so I would add my vote to making it (parsing) as easy and
> consistent as possible.

Tony - I warned/suggested last week that we _may_ find that we need the
same info dumped out in different formats for Tcl, perl, [c]sh, etc. It's
not clear yet whether or not this is reequired but Bob wasn't against the
idea.

...RickM...

Re: proclamation of intent

[Bob Manson]

In message < Pine.GSO.4.10.9908231158010.20253-100000@sys4 >, Rick Macdonald writ
es:
>The fields and enums should have human-readable Help strings (multi-line
>allowed?), and these strings need to be available to the client.

Agreed. I haven't done this yet but it's a trivial thing to add; I'll
probably just put it to the grammar after I'm done with this message.
Nothing will actually use it yet.

>I would like that all configuration stuff is read and parsed by GNATS, and
>when sent to the client, it is formatted and written by GNATS (gnatsd)
>such that the client just parses it and doesn't have to check for errors

I had little intention of just dumping out a config file, although it
may make a bit of sense to do this for the network protocol.  The
parser's already in the network client, so there's little to be gained
to split it all apart in N different ways.  But none of this is
"written in stone"; I'm certainly open to other people's suggestions!

I honestly don't have any idea what the configuration info as sent to
clients will look like.  I think now would be a great time for people
to make suggestions as to what they'd like to see.

The info that needs to be presented is really in two parts.  The first part
will simply be field definitions, including but not limited to:

	name
	(optional) internal field designation
	type of data stored in field
	additional per-data-type options:
		enum list
		regexp qualifier
		default match type
	default value
	help string

name:	The name of the field. This will not include the '>' or ':' separator
	characters.  This may be different from the name used in the PR
	to mark the field (and if so, the PR field string would need to
	be included as well).

internal field designation: Some fields are internally used by GNATS, such
	as Category, PR Number, etc.  There needs to be a mechanism to
	relate these internal fields to their actual definitions.

type of data:  text, enum, date, multitext.  Some text fields may be
	further qualified by a regexp that they must match.

	I've given some thought to adding a "numeric" type as well, and I 
	think it's essential to limit this basic information to a manageable
	set, so any other ideas need to be implemented soon.

per-data-type options:  This should be fairly obvious; for an enum, the
	possible values need to be listed, etc.

	"default match type" simply refers to the type of search that is done
	by default.  Currently some fields in GNATS are set up as "must match
	entire field" when a regexp is used on that field, while others are
	"may match any part of field".

default value: Obvious.

help string: A human-readable description of the field.

The second part would be the actual "presentation format" of the PR,
at least in terms of field order.  This may optionally include a
template such as that used by edit-pr, although this would be fairly
useless for a web-based client; it might also include suggested field
headings for a forms-based client.  I'm leaving this part rather
vague, since I don't honestly know what would be most useful.

Again, I am certainly not proposing any particular format; I'm totally
open to suggestions for this.  I'm also trying very hard to not break
existing clients that have hard-coded field information.
						Bob

client info format ideas

[Rick Macdonald]

On Mon, 23 Aug 1999, Bob Manson wrote:

> I honestly don't have any idea what the configuration info as sent to
> clients will look like.  I think now would be a great time for people
> to make suggestions as to what they'd like to see.

OK, here's a rough suggestion, just to get some reactions. I've grabbed
these examples from your previous post:

        field "Release-Note" {
                multitext
                query-default exact-regexp
        }

        field "Release" {
                string matching "insert-regex-here"
                query-pr-option "A" "--release"
                network-query-command "RLSE"
                query-default inexact-regexp
        }

        field "Priority" {
                enum {
                        values {
                                "high" "medium" "low"
                        }
                        default "medium"
                }
                query-pr-option "p" "--priority"
                network-query-command "PRIO"
                query-default exact-regexp
        }

Looking now, wouldn't it be more straight-forward if all lines were
keyword-value pairs?

What's missing is that type-of-data has no keyword. It could be:

        field "Release-Note" {
                type multitext
and   
        field "Priority" {
		type enum
                enum { ...
  
and                      
                        
        field "Release" {
                type text
                stringmatching "insert-regex-here"

I'm not sure about your "string matching". Are there other sub options to
"string" other than "matching"? If not, that's why I combined it to one
keyword "stringmatching".

The point of this little cleanup suggestion is to facilitate this format:

fieldname "list of keyword-value pairs"

(I realize that the format of the gnats config files doesn't have to be
exactly like the info sent to clients, but it might as well start off
being more similar, I figure!)

Tcl and perl have different list syntax, but the following idea might be
useful for perl as well(?).

For Tcl:

fieldname {keyword {value} keyword {value} keyword {value}}

So, for the fields above:

Release-Note {type {multitext} query-default {exact-regexp}}

Release {type {text} stringmatching {[0-9]\.[0-9]} query-pr-option {A
--release} network-query-command {RLSE} query-default {inexact-regexp}}

Priority {type {enum} enum {values {high medium low} default {medium}}
query-pr-option {p --priority} network-query-command {PRIO} query-default
{exact-regexp}}

With this format, I don't even have to parse:

If variable "f" is set to the "Priority" line above:

% array set keys [lindex $f 1]
% array names keys
query-default network-query-command enum type query-pr-option

and now there's an array called "keys" with elements equal to your
keywords.

Note that "enum" is nested:

% puts $keys(enum)
values {high medium low} default {medium}

but I can do the same thing:

% array set enumkeys $keys(enum)
% array names enumkeys
values default
% puts $enumkeys(values)
high medium low

There may be drawbacks with the above, such as restricions on the number
of characters in one line (before an LF or CR/LF). Also, the brace
characters, if they exist in the keyword-values, would have to be escaped.

An alternative might be just a stream of records:

220 List follows.
field Priority
type enum
enumvalues high medium low
enumdefault medium
enumquery-pr-option p --priority
network-query-command PRIO 
query-default exact-regexp
field Release
...etc...
query-default inexact-regexp
.

In this case I've "flattened" the nested enum into the one level of
keyword-value pairs. It's not really much more work to read this and build
my own lists or arrays.

...RickM...

Re: proclamation of intent

[Mike Sutton]

>>>>> "Paul" == Paul Traina <pst@juniper.net> writes:

    > If we're backwards compatible with 3.x's API's and GNATS socket
    > interface, I'm cool with calling it 3.300.  However I note that
    > we have a unique opportunity to clean up some of the additional
    > configuration mess that has occured over the years (e.g. gnatsd
    > config files are a nightmare) and if we want to make backwards
    > incompatible changes, we should do a 4.0 beta release.

I vote for a 4.0 release.  Lots of things have changed since 3.2.
This would also provide a nice clean break point where some of the
backward compatibilities could be dropped.

My $0.02


Mike Sutton                      | public class
SAIC				 | software_failure : management_failure
Beavercreek, OH			 | 
Suttonm@saic.com                 | These are MY opinions, not SAIC's

Re: proclamation of intent

[Mike Sutton]

>>>>> "Rick" == Rick Macdonald <rickm@vsl.com> writes:

    > The fields and enums should have human-readable Help strings
    > (multi-line allowed?), and these strings need to be available to
    > the client.

    > I would like that all configuration stuff is read and parsed by
    > GNATS, and when sent to the client, it is formatted and written
    > by GNATS (gnatsd) such that the client just parses it and
    > doesn't have to check for errors for remove comments. By this I
    > mean that GNATS shouldn't just copy a gnats admin file to the
    > output (gnats used to do this; I removed a few cases of
    > it). This also makes the formatting of the info more consistent
    > for parsing by clients.

I agree here.  The less verifing that the front ends have to do the
better.  Gnats should make sure everything is correct and then send
just what is needed across the net.



Mike Sutton                      | public class
SAIC				 | software_failure : management_failure
Beavercreek, OH			 | 
Suttonm@saic.com                 | These are MY opinions, not SAIC's

Re: proclamation of intent

[Michael Richardson]

>>>>>> "Mike" == Mike Sutton <mws115@llcoolj.dayton.saic.com> writes:
    Mike> I vote for a 4.0 release.  Lots of things have changed since 3.2.
    Mike> This would also provide a nice clean break point where some of the
    Mike> backward compatibilities could be dropped.
  
  I definitely agree. It has been too long. Call it 4.0

   :!mcr!:            |  Solidum Systems Corporation, http://www.solidum.com
   Michael Richardson |For a better connected world,where data flows faster<tm>
 Personal: <A HREF=" http://www.sandelman.ottawa.on.ca/People/Michael_Richardson/Bio.html ">mcr@sandelman.ottawa.on.ca</A>. PGP key available.
 Corporate: <A HREF=" mailto:mcr@solidum.com ">mcr@solidum.com</A>.