RE: modular database backends

RE: modular database backends

[Dirk Bergstrom]

a database backend for gnats, huh?  i've been thinking about this myself.
in fact, it's quite likely to be a major project for me this year.  i
haven't gotten too far along, but i will soon have a lot more time to devote
to thinking about it.  i wouldn't describe myself as a high powered C
programmer (cough cough...), but i've got clue, time, and energy.

we should gather together interested parties, and kick some ideas around.

--
Dirk Bergstrom              dirk@juniper.net
____________________________________________
Juniper Networks Inc.,  Engineering Web Guru
Tel: 408.745.3182          Fax: 408.745.8905


> -----Original Message-----
> From: Peter Novodvorsky [ mailto:nidd@altlinux.ru ]
> Sent: Sunday, May 27, 2001 11:18 AM
> To: gnats-devel@sources.redhat.com
> Subject: modular database backends
> 
> 
> 
> Hello!
> 
>         I always thought about BTS as database and tools to work with
> it. And these things are different. In gnats I see opposite 
> thing: database,
> tools and servers share one source and tools sometimes do too much.
> 
>         So I've began work on making database backends to 
> gnats modular. 
> Just now, gnats has two "backends", network (gnatsd) and filsystem. If
> we make system of backends modular we could have SQL backend, 
> etc., etc.
> 
>         Second thing I don't like in  GNATS, it doesn't has ACLs. This
> modular framework should support them.  Here is the header file that
> will briefly describe what I mean:
> 
> struct _gnats_database {
>   URI *U;
>   
>   FILE *serv_read;
>   FILE *serv_write;
> 
>   void *module_handle;
> 
>   int (* init) (ErrorDesc *err, gnats_database);
>   int (* add_pr) (ErrorDesc *err, gnats_database, PR);
>   int (* update_pr) (ErrorDesc *err, gnats_database, PR);
>   int (* query_pr) (ErrorDesc *err, gnats_database, QueryExpr);
>   int (* disconnect) (ErrorDesc *err, gnats_database);
> };
> 
> typedef struct _gnats_database *gnats_database;
> 
> gnats_database gnats_new_database (ErrorDesc *err, URI *U);
> int gnats_destroy_database (gnats_database db);
> 
>         module_handle is object handle that is returned by dlopen.
> URI is:
> 
> struct _URI
> {
>   char *scheme;
>   char *host;
>   char *user;
>   char *pwd;
>   char *path;
>   unsigned int port;
> };
> 
> typedef struct _URI URI;
> 
>         And there is a function
> 
> URI *gnats_parse_uri (const char *gnats_uri, int length);
> 
> that takes URI (for example 
> gnatsd4x://nidd:pwd@nidd.debian.net:1529/database_name) and converts
> it in such structure (gnatsd4x is 4.0 gnatsd protocol).
> 
>         I'm currently implementing gnatsd4x backend. After this I'll
> make query-pr, pr-edit, gnatsd and other tools to work with 
> this model.
> And then I'll begin to make SQL backend.
> 
>         If everything will be OK in the end following will be 
> possible:
> 
>            user -> client -> gnatsd -> SQL server (MySQL, Postgres)
> 
>         What do you think of this crazy idea?
> 
>                                                 Thanks,
> -- 
> Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
> Debian.Org                                       
http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

Re: modular database backends

[Bob Kaehms]

Just as a thought, perhaps the whole thing should be done in php/mysql.
it's a little archaic as is.
Dirk Bergstrom writes: 

> a database backend for gnats, huh?  i've been thinking about this myself.
> in fact, it's quite likely to be a major project for me this year.  i
> haven't gotten too far along, but i will soon have a lot more time to devote
> to thinking about it.  i wouldn't describe myself as a high powered C
> programmer (cough cough...), but i've got clue, time, and energy. 
> 
> we should gather together interested parties, and kick some ideas around. 
> 
> --
> Dirk Bergstrom              dirk@juniper.net
> ____________________________________________
> Juniper Networks Inc.,  Engineering Web Guru
> Tel: 408.745.3182          Fax: 408.745.8905 
> 
> 
>> -----Original Message-----
>> From: Peter Novodvorsky [ mailto:nidd@altlinux.ru ]
>> Sent: Sunday, May 27, 2001 11:18 AM
>> To: gnats-devel@sources.redhat.com
>> Subject: modular database backends 
>> 
>>  
>> 
>> Hello! 
>> 
>>         I always thought about BTS as database and tools to work with
>> it. And these things are different. In gnats I see opposite 
>> thing: database,
>> tools and servers share one source and tools sometimes do too much. 
>> 
>>         So I've began work on making database backends to 
>> gnats modular. 
>> Just now, gnats has two "backends", network (gnatsd) and filsystem. If
>> we make system of backends modular we could have SQL backend, 
>> etc., etc. 
>> 
>>         Second thing I don't like in  GNATS, it doesn't has ACLs. This
>> modular framework should support them.  Here is the header file that
>> will briefly describe what I mean: 
>> 
>> struct _gnats_database {
>>   URI *U;
>>   
>>   FILE *serv_read;
>>   FILE *serv_write; 
>> 
>>   void *module_handle; 
>> 
>>   int (* init) (ErrorDesc *err, gnats_database);
>>   int (* add_pr) (ErrorDesc *err, gnats_database, PR);
>>   int (* update_pr) (ErrorDesc *err, gnats_database, PR);
>>   int (* query_pr) (ErrorDesc *err, gnats_database, QueryExpr);
>>   int (* disconnect) (ErrorDesc *err, gnats_database);
>> }; 
>> 
>> typedef struct _gnats_database *gnats_database; 
>> 
>> gnats_database gnats_new_database (ErrorDesc *err, URI *U);
>> int gnats_destroy_database (gnats_database db); 
>> 
>>         module_handle is object handle that is returned by dlopen.
>> URI is: 
>> 
>> struct _URI
>> {
>>   char *scheme;
>>   char *host;
>>   char *user;
>>   char *pwd;
>>   char *path;
>>   unsigned int port;
>> }; 
>> 
>> typedef struct _URI URI; 
>> 
>>         And there is a function 
>> 
>> URI *gnats_parse_uri (const char *gnats_uri, int length); 
>> 
>> that takes URI (for example 
>> gnatsd4x://nidd:pwd@nidd.debian.net:1529/database_name) and converts
>> it in such structure (gnatsd4x is 4.0 gnatsd protocol). 
>> 
>>         I'm currently implementing gnatsd4x backend. After this I'll
>> make query-pr, pr-edit, gnatsd and other tools to work with 
>> this model.
>> And then I'll begin to make SQL backend. 
>> 
>>         If everything will be OK in the end following will be 
>> possible: 
>> 
>>            user -> client -> gnatsd -> SQL server (MySQL, Postgres) 
>> 
>>         What do you think of this crazy idea? 
>> 
>>                                                 Thanks,
>> -- 
>> Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
>> Debian.Org                                       
> http://debian.org/~nidd
>             Debian  ---  no need to  wait for tomorrow.
 

Re: modular database backends

[Peter Novodvorsky]

> Just as a thought, perhaps the whole thing should be done in php/mysql.
> it's a little archaic as is.

If you say so, you didn't got the idea. I don't want  only one backend.
I want modular backends.

> > a database backend for gnats, huh?  i've been thinking about this myself.
> > in fact, it's quite likely to be a major project for me this year.  i
> > haven't gotten too far along, but i will soon have a lot more time to devote
> > to thinking about it.  i wouldn't describe myself as a high powered C
> > programmer (cough cough...), but i've got clue, time, and energy. we
> > should gather together interested parties, and kick some ideas

What do you  think about  my prototype? Is it good or bad? I'm  going
to do modules through dlopen. I would  like to hear you thoughts about it.


Thanks,

-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

gnatsweb

[Margaret BRIERTON]

Hi

Can someone tell me what the permissions ie owner and group should be for the
file bugs.log?

I get as far as submitting a problem report using gnatsweb and i get the
following error:


/usr/local/share/gnats/gnats-db/gnats-adm/bugs.log... Can't create output:
Permission denied
/dead.letter... cannot open /dead.letter: Permission denied
 Error


Bad pipe to /usr/lib/sendmail -oi -t


I created the file as the GNATS admin.


cheers

margaret

Re: gnatsweb

[S Ramesh]

Hi Margaret,

Check out ur /etc/aliases file.
In that u should have a alias like

bug-log: /usr/local/share/gnats/gnats-db/gnats-adm/bugs.log

if its so while submitting a bug.. ur mail will create that bugs.log and append
all the bugs then and there. tthe owner and group will be "mail".
u can view that file as root.

This is working for me. BUt i have another problem.
the previous alias like

bug -q: /usr/local/libexec/gnats/queue-pr

is not woring for me. the mail is returning and its saying that int he returned
mail that

queue-pr service not availabe like that. can be plz help me to sort it out. this
gnats installation is taking me days. Also i have apache webserver on the the
same host where gnats is installed. i installed gnatsweb likw

make install CGI_DIR=/var/www/cgi-bin.

it is installed there.. but how to invoke it.. when i put my IP addr on the
browser.. ts displaying the deauflt test page only. i have gnatsd running
.Kindly help me.. ia m in urgent need of these things.. which i got stuck two
days back.

Regds,
Ramesh

******************************************************************************88

Margaret BRIERTON wrote:

> Hi
>
> Can someone tell me what the permissions ie owner and group should be for the
> file bugs.log?
>
> I get as far as submitting a problem report using gnatsweb and i get the
> following error:
>
> /usr/local/share/gnats/gnats-db/gnats-adm/bugs.log... Can't create output:
> Permission denied
> /dead.letter... cannot open /dead.letter: Permission denied
>  Error
>
> Bad pipe to /usr/lib/sendmail -oi -t
>
> I created the file as the GNATS admin.
>
> cheers
>
> margaret

modular database backends

[Peter Novodvorsky]

Hello!

        I always thought about BTS as database and tools to work with
it. And these things are different. In gnats I see opposite thing: database,
tools and servers share one source and tools sometimes do too much.

        So I've began work on making database backends to gnats modular. 
Just now, gnats has two "backends", network (gnatsd) and filsystem. If
we make system of backends modular we could have SQL backend, etc., etc.

        Second thing I don't like in  GNATS, it doesn't has ACLs. This
modular framework should support them.  Here is the header file that
will briefly describe what I mean:

struct _gnats_database {
  URI *U;
  
  FILE *serv_read;
  FILE *serv_write;

  void *module_handle;

  int (* init) (ErrorDesc *err, gnats_database);
  int (* add_pr) (ErrorDesc *err, gnats_database, PR);
  int (* update_pr) (ErrorDesc *err, gnats_database, PR);
  int (* query_pr) (ErrorDesc *err, gnats_database, QueryExpr);
  int (* disconnect) (ErrorDesc *err, gnats_database);
};

typedef struct _gnats_database *gnats_database;

gnats_database gnats_new_database (ErrorDesc *err, URI *U);
int gnats_destroy_database (gnats_database db);

        module_handle is object handle that is returned by dlopen.
URI is:

struct _URI
{
  char *scheme;
  char *host;
  char *user;
  char *pwd;
  char *path;
  unsigned int port;
};

typedef struct _URI URI;

        And there is a function

URI *gnats_parse_uri (const char *gnats_uri, int length);

that takes URI (for example 
gnatsd4x://nidd:pwd@nidd.debian.net:1529/database_name) and converts
it in such structure (gnatsd4x is 4.0 gnatsd protocol).

        I'm currently implementing gnatsd4x backend. After this I'll
make query-pr, pr-edit, gnatsd and other tools to work with this model.
And then I'll begin to make SQL backend.

        If everything will be OK in the end following will be possible:

           user -> client -> gnatsd -> SQL server (MySQL, Postgres)

        What do you think of this crazy idea?

                                                Thanks,
-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

modular database backends

[Peter Novodvorsky]

Hello!

        I always thought about BTS as database and tools to work with
it. And these things are different. In gnats I see opposite thing: database,
tools and servers share one source and tools sometimes do too much.

        So I've began work on making database backends to gnats modular. 
Just now, gnats has two "backends", network (gnatsd) and filsystem. If
we make system of backends modular we could have SQL backend, etc., etc.

        Second thing I don't like in  GNATS, it doesn't has ACLs. This
modular framework should support them.  Here is the header file that
will briefly describe what I mean:

struct _gnats_database {
  URI *U;
  
  FILE *serv_read;
  FILE *serv_write;

  void *module_handle;

  int (* init) (ErrorDesc *err, gnats_database);
  int (* add_pr) (ErrorDesc *err, gnats_database, PR);
  int (* update_pr) (ErrorDesc *err, gnats_database, PR);
  int (* query_pr) (ErrorDesc *err, gnats_database, QueryExpr);
  int (* disconnect) (ErrorDesc *err, gnats_database);
};

typedef struct _gnats_database *gnats_database;

gnats_database gnats_new_database (ErrorDesc *err, URI *U);
int gnats_destroy_database (gnats_database db);

        module_handle is object handle that is returned by dlopen.
URI is:

struct _URI
{
  char *scheme;
  char *host;
  char *user;
  char *pwd;
  char *path;
  unsigned int port;
};

typedef struct _URI URI;

        And there is a function

URI *gnats_parse_uri (const char *gnats_uri, int length);

that takes URI (for example 
gnatsd4x://nidd:pwd@nidd.debian.net:1529/database_name) and converts
it in such structure (gnatsd4x is 4.0 gnatsd protocol).

        I'm currently implementing gnatsd4x backend. After this I'll
make query-pr, pr-edit, gnatsd and other tools to work with this model.
And then I'll begin to make SQL backend.

        If everything will be OK in the end following will be possible:

           user -> client -> gnatsd -> SQL server (MySQL, Postgres)

        What do you think of this crazy idea?

                                                Thanks,
-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

Re: modular database backends

[Yngve Svendsen]

At 22:18 27.05.2001 +0400, Peter Novodvorsky wrote:
>        I'm currently implementing gnatsd4x backend. After this I'll
>make query-pr, pr-edit, gnatsd and other tools to work with this model.
>And then I'll begin to make SQL backend.
>
>         If everything will be OK in the end following will be possible:
>
>            user -> client -> gnatsd -> SQL server (MySQL, Postgres)
>
>         What do you think of this crazy idea?

I really support this wholeheartedly, and I know that a lot of people would 
welcome this feature very much.

Yngve Svendsen
IS Engineer
Clustra AS, Trondheim, Norway
yngve.svendsen@clustra.com

Re: modular database backends

[Peter Novodvorsky]

Yngve Svendsen <yngve.svendsen@clustra.com> writes:

> At 22:18 27.05.2001 +0400, Peter Novodvorsky wrote:
> >        I'm currently implementing gnatsd4x backend. After this I'll
> >make query-pr, pr-edit, gnatsd and other tools to work with this model.
> >And then I'll begin to make SQL backend.
> >
> >         If everything will be OK in the end following will be possible:
> >
> >            user -> client -> gnatsd -> SQL server (MySQL, Postgres)
> >
> >         What do you think of this crazy idea?
> 
> I really support this wholeheartedly, and I know that a lot of people
> would welcome this feature very much.

I'm not professional programmer and I would like to discuss implementation
of this feature with  maintainer. In fact this feature requires quite a
lot of hacking,  and it  can  happen that Milan won't like it.


                                                Thanks,
-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

Re: modular database backends

[Margaret BRIERTON]

Hi

Can anyone tell me if there is a GNATS discussion group.  I'm having a few
problems getting it up and running on my machine and would really appreciate a
bit of input into where i'm going wrong.

Does anyone know if such a group exists?

Cheers

Margaret

Re: modular database backends

[Milan Zamazal]

>>>>> "PN" == Peter Novodvorsky <nidd@altlinux.ru> writes:

    PN>         So I've began work on making database backends to gnats
    PN> modular.  Just now, gnats has two "backends", network (gnatsd)
    PN> and filsystem. If we make system of backends modular we could
    PN> have SQL backend, etc., etc.

Yes, this is a wanted feature.

    PN>         Second thing I don't like in GNATS, it doesn't has
    PN> ACLs. This modular framework should support them.  Here is the
    PN> header file that will briefly describe what I mean:

I'm afraid I couldn't understand what kind of ACLs you would like to
have.  Could you elaborate?

    PN>         If everything will be OK in the end following will be
    PN> possible:

    PN>            user -> client -> gnatsd -> SQL server (MySQL,
    PN> Postgres)

    PN>         What do you think of this crazy idea?

I can't see anything crazy in it. :-) Actually I was already asked by my
friend about the possibility of using LDAP as the PR storage.

However, no new features will get into GNATS 4.0.  I think the backend
architecture and the MIME handling could be the main feature candidates
for the first non-bug-fix release after 4.0.  Of course, we can start
discussing the API and architecture now and you can also work on a
preliminary implementation in the meantime, just don't expect it to have
it in 4.0.

    PN> I'm not professional programmer and I would like to discuss
    PN> implementation of this feature with maintainer.

OK.

BTW, please note I'll be soon out of Internet for about a week or two.

Regards,

Milan Zamazal

-- 
Omigod, it's a flame war about a flame war.  You know, a meta-flame war!
                                                 Kenny Tilton in comp.lang.lisp

Re: modular database backends

[Peter Novodvorsky]

Milan Zamazal <pdm@zamazal.org> writes:

> >>>>> "PN" == Peter Novodvorsky <nidd@altlinux.ru> writes:
> 
>     PN>         Second thing I don't like in GNATS, it doesn't has
>     PN> ACLs. This modular framework should support them.  Here is the
>     PN> header file that will briefly describe what I mean:
> 
> I'm afraid I couldn't understand what kind of ACLs you would like to
> have.  Could you elaborate?

1). i don't like that client can pass without authentication phase.
If he want to login as anonymous,  he  has to do it. IE,

USER anonymous nidd@altlinux.ru

2). I don't like that check of user/password is made by gnatsd and not
by "database driver". I mean that function that authentificates user
isn't called by init_gnats that actually connects to database, but after
it.  Access control is done by gnatsd, and not by database driver.

3). There is no access control for query-pr, send-pr and edit-pr. This 
is close to the subject of 2)., because  authentification is made on
the side of client, not on the side of database.

Advanced topics.

4). I want that some bugs  can be edited only by group of developers.
It can't be done in current version.

Is it enough? ;-)


> However, no new features will get into GNATS 4.0.  I think the backend
> architecture and the MIME handling could be the main feature candidates
> for the first non-bug-fix release after 4.0.  Of course, we can start
> discussing the API and architecture now and you can also work on a
> preliminary implementation in the meantime, just don't expect it to have
> it in 4.0.

May be you could make a CVS branch for 4.99?

> BTW, please note I'll be soon out of Internet for about a week or two.

Heh, that's bad because I wanted to begin making implementation very
actively. Do you have thoughts on the subject?  I'm writing
``specification'' of some kind. Maybe it   will give the subject for
discussion.

Regards,

-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

Re: modular database backends

[Milan Zamazal]

>>>>> "PN" == Peter Novodvorsky <nidd@altlinux.ru> writes:

    PN> May be you could make a CVS branch for 4.99?

Not yet.  That would require a lot of duplicate commits in the bug
fixing and cleanup phase.

    PN> Heh, that's bad because I wanted to begin making implementation
    PN> very actively. Do you have thoughts on the subject?  I'm writing
    PN> ``specification'' of some kind. Maybe it will give the subject
    PN> for discussion.

Well, the first ideas are:

- A function for deleting a PR should be added.  This is only seldom
  used, but it's sometimes useful and can also be used e.g. by
  maintenance tools.

- Maybe it's not *necessary* to force backends to implement full query
  handling.  We probably can agree on that writing new backends should
  be as simple as possible.  I can imagine that your `query_pr' function
  could be only optional and there could be available simpler versions
  of query functions that can get a list of all problem IDs, a
  particular PR and maybe also optionally some index (similar to the
  current one).

Regards,

Milan Zamazal

-- 
  _/_\_/_  o     _\_/_\_  o     _/_\_/_  o     _\_/_\_  o    BEWARE!
-<_|_|_|_><--  -<_|_|_|_><--  -<_|_|_|_><--  -<_|_|_|_><--   *Bugs* are
   / \ /   o      \ / \   o      / \ /   o      \ / \   o     approaching!

Re: modular database backends

[Peter Novodvorsky]

Milan Zamazal wrote:

>     PN> May be you could make a CVS branch for 4.99?
> 
> Not yet.  That would require a lot of duplicate commits in the bug
> fixing and cleanup phase.
> 
>     PN> Heh, that's bad because I wanted to begin making implementation
>     PN> very actively. Do you have thoughts on the subject?  I'm writing
>     PN> ``specification'' of some kind. Maybe it will give the subject
>     PN> for discussion.
> 
> Well, the first ideas are:
> 
> - A function for deleting a PR should be added.  This is only seldom
>   used, but it's sometimes useful and can also be used e.g. by
>   maintenance tools.

Yeah, this is reasonable.

> 
> 
> - Maybe it's not *necessary* to force backends to implement full query
>   handling.  We probably can agree on that writing new backends should
>   be as simple as possible.  I can imagine that your `query_pr' function
>   could be only optional and there could be available simpler versions
>   of query functions that can get a list of all problem IDs, a
>   particular PR and maybe also optionally some index (similar to the
>   current one).

We can implement simplier functions in library that will open this 
module. And
these functions will use query_pr as more low level function. Any 
objections?

Nidd.

Re: modular database backends

[Milan Zamazal]

>>>>> "PN" == Peter Novodvorsky <nidd@altlinux.ru> writes:

    PN> Milan Zamazal wrote:
    >> - Maybe it's not *necessary* to force backends to implement full
    >> query handling.  We probably can agree on that writing new
    >> backends should be as simple as possible.  I can imagine that
    >> your `query_pr' function could be only optional and there could
    >> be available simpler versions of query functions that can get a
    >> list of all problem IDs, a particular PR and maybe also
    >> optionally some index (similar to the current one).

    PN> We can implement simplier functions in library that will open
    PN> this module. And these functions will use query_pr as more low
    PN> level function. Any objections?

I'm not sure we understand each other.  To clarify the things: For
instance, we could make a backend library that defines three functions
query_pr, get_list_of_pr_ids, get_pr.  query_pr is implemented in the
library using the other two functions and those are implemented using
query_pr.  Every backend must provide its own implementation of either
query_pr or get_list_of_pr_ids and get_pr, the other functions may (but
needn't) be implemented using the library.

So a backend powered by a sophisticated query engine (e.g. SQL) can
implement the complex query_pr function itself and needn't bother to
implement the simpler functions get_*.  On the other hand, a file system
backend would implement the get_* functions and use the library function
for query_pr.

OK?

Milan Zamazal

-- 
And why?