RE: persistent DB connections (was: RE: modular database backends)

RE: persistent DB connections (was: RE: modular database backends)

[Dirk Bergstrom]

i, too, am inclined towards #3, but it does require rather a lot of
re-engineering.  under the current setup, gnatsd generally runs for only one
"transaction" (a search, a pr-edit, one gnatsweb hit, etc.).  turning it
into a persistent daemon would, to put it nicely, help us discover any
hidden memory leaks.

furthermore, while gnatsd does have the RSET command ("reset internal
settings to initial startup"), it doesn't have any of the other mechanisms
necessary to a persistent daemon.  i'm sure the code is readily available,
but it's not trivial to get everything working, stable, and leak-free...

--
Dirk Bergstrom              dirk@juniper.net
____________________________________________
Juniper Networks Inc.,  Engineering Web Guru
Tel: 408.745.3182          Fax: 408.745.8905


> -----Original Message-----
> From: Michael Richardson [ mailto:mcr@sandelman.ottawa.on.ca ]
> Sent: Wednesday, May 30, 2001 9:09 AM
> To: Dirk Bergstrom
> Subject: Re: persistent DB connections (was: RE: modular database
> backends) 
> 
> 
> 
>   You should put the database connection in gnatsd.
>   It does not have to be spawned from inetd.
> 
>   #3 is where you should go.
> 
> Canadian Commuter Challenge Project -- GNU Potato Caboose 
> Michael Richardson, Sandelman Software Works, Ottawa, ON  
> EMAIL: mcr@commuterchallenge.net
> for help, email or page at 1-866-231-8608
> 
> 

Re: persistent DB connections (was: RE: modular database backends )

[Michael Richardson]

>>>>> "Dirk" == Dirk Bergstrom <dirk@juniper.net> writes:
    Dirk> i, too, am inclined towards #3, but it does require rather a lot of
    Dirk> re-engineering.  under the current setup, gnatsd generally runs for only one
    Dirk> "transaction" (a search, a pr-edit, one gnatsweb hit, etc.).  turning it
    Dirk> into a persistent daemon would, to put it nicely, help us discover any
    Dirk> hidden memory leaks.

  Not necessarily.
  The "daemon" may consist of doing:

      1. open connection to database
      2. listen
      3. accept, fork process connection.
      4. loop

  + SIGCLD routine.
 
  The only thing that is being "preserved" is the connection to the database.
This depends upon the connection being one that can be shared. If it can't
be shared, then you get more into:
   1. open connection(s) to database
   2. listen
   3. while connection(s) available:
      accept, grab connection, fork, process
   4. loop
   + SIGCLD to return connection to available list.

Canadian Commuter Challenge Project -- GNU Potato Caboose 
Michael Richardson, Sandelman Software Works, Ottawa, ON  
EMAIL: mcr@commuterchallenge.net
for help, email or page at 1-866-231-8608

Re: persistent DB connections (was: RE: modular database backends)

[Milan Zamazal]

>>>>> "DB" == Dirk Bergstrom <dirk@juniper.net> writes:

    DB> 1) keep the current model, adding a separate, persistent,
    DB> "database daemon".  each connection would spawn a new instance
    DB> of gnatsd, which would connect to the database-daemon as needed.
    DB> i happened across sqlrelay
    DB> ( http://www.firstworks.com/sqlrelay.html ), a GPLed package which
    DB> appears to do just what's needed for the database daemon.

Since the addition the modular database backends feature would allow
this automatically, there'll be nothing to stop you doing this once it
is implemented.

    DB> 2) a persistent daemon that handles setting up the connection to
    DB> the client, and holds open a connection (or several connections)
    DB> to the database.  as new requests come in, the daemon would fork
    DB> off a child to handle each one, up to some preset limit.

I can imagine a very busy BTS where performance problems can arise.  If
you can *prove* that the overhead of reading configuration is a
significant part of the performance problem, I'll consider the
possibility of making a persistent daemon.

    DB> 3) something like apache -- a master process forks several
    DB> persistent daemons, which take turns handling connections.  each
    DB> child daemon holds a database connection, and services many
    DB> requests before shutting down.

IMHO unnecessarily complicated:

>>>>> "PN" == Peter Novodvorsky <nidd@altlinux.ru> writes:

    PN> Other approaches are good, but hey, do some BTS's handle more
    PN> then 10 connections at time? Really?

AOL!

Regards,

Milan Zamazal

-- 
  _/_\_/_  o     _\_/_\_  o     _/_\_/_  o     _\_/_\_  o    BEWARE!
-<_|_|_|_><--  -<_|_|_|_><--  -<_|_|_|_><--  -<_|_|_|_><--   *Bugs* are
   / \ /   o      \ / \   o      / \ /   o      \ / \   o     approaching!

Re: persistent DB connections (was: RE: modular database backends)

[Peter Novodvorsky]

Hello!

You're speaking about _new_ backend daemon. We could make a lot of backend
daemons, but it is second stage, IMO. 

Any daemon has fixed language (protocol) in input and modular in output. 
What you are talking  currently is rewrite of GNATSD.

Dirk Bergstrom <dirk@juniper.net> writes:

> 1) keep the current model, adding a separate, persistent, "database daemon".
> each connection would spawn a new instance of gnatsd, which would connect to
> the database-daemon as needed.  i happened across sqlrelay
> ( http://www.firstworks.com/sqlrelay.html ), a GPLed package which appears to
> do just what's needed for the database daemon.

This is most simple, and IMO, first that should be made. 

> 
> 2) a persistent daemon that handles setting up the connection to the client,
> and holds open a connection (or several connections) to the database.  as
> new requests come in, the daemon would fork off a child to handle each one,
> up to some preset limit.
> 
> 3) something like apache -- a master process forks several persistent
> daemons, which take turns handling connections.  each child daemon holds a
> database connection, and services many requests before shutting down.

Other approaches are good, but hey, do some BTS's handle more then 10 
connections  at  time? Really?

        Thanks,
-- 
Peter Novodvorsky     http://www.altlinux.ru/    AltLinux Team, Russia
Debian.Org                                       http://debian.org/~nidd
            Debian  ---  no need to  wait for tomorrow.

persistent DB connections (was: RE: modular database backends)

[Dirk Bergstrom]

one of my main reasons to move to a database backend is performance &
scaling; setting up & tearing down a database connection for every gnats
transaction is just not the way to make a speedy & scalable application.  i
believe that a database-backed gnats will need some way to maintain a
persistent connection to that database.

currently gnatsd is spawned by inetd as needed, and it exits after
performing a single transaction (generally).  here's a few ways we could
handle the database connection:

1) keep the current model, adding a separate, persistent, "database daemon".
each connection would spawn a new instance of gnatsd, which would connect to
the database-daemon as needed.  i happened across sqlrelay
( http://www.firstworks.com/sqlrelay.html ), a GPLed package which appears to
do just what's needed for the database daemon.

2) a persistent daemon that handles setting up the connection to the client,
and holds open a connection (or several connections) to the database.  as
new requests come in, the daemon would fork off a child to handle each one,
up to some preset limit.

3) something like apache -- a master process forks several persistent
daemons, which take turns handling connections.  each child daemon holds a
database connection, and services many requests before shutting down.

#1 has the advantage of being most like the current codebase, and would thus
require the least work.  this would also best support a modular approach --
gnats could be installed with a database-daemon module, or the current
flat-file module.  on the other hand, each client connection has to go
through the overhead of reading in all the information about a database
(dbconfig, categories, responsible, gnatsd.access, etc.).  it's not a lot of
overhead, but i find it vaguely offensive to build something anew every
couple seconds...

#2 and #3 are fairly similar, as both allow for caching of the difficult
stuff.  i'm not sure which would be better -- i'd be inclined to say #2, but
i can't ignore the fact that apache works like #3, and it's a *very*
successful system.  in any case i think i like both of these more than #1,
though they require more rewriting.

what do y'all think?

--
Dirk Bergstrom              dirk@juniper.net
____________________________________________
Juniper Networks Inc.,  Engineering Web Guru
Tel: 408.745.3182          Fax: 408.745.8905