fix for PR 219 - large PRs or updates take many minutes to process

fix for PR 219 - large PRs or updates take many minutes to process

[Dirk Bergstrom]

after i submitted PR 219, it bit us again, locking up the server for a
couple hours.  i decided i'd apply my meager C skills, and see if i
couldn't fix it.  it seemed clear it was a badly written loop, and it
wasn't too hard to find it.  turns out there were two problems, a typo
and a needlessly repeated strlen() call.  the patch gives a 1000x
improvement in processing time for large PR updates.

the PR is here:

http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&pr=219&database=g
nats

and the patch is here (untouched by meddlesome MTAs):

http://otisbean.com/file-pr_patch.txt

and here it is for your reading pleasure:

Index: file-pr.c
===================================================================
RCS file: /cvs/gnats/gnats/gnats/file-pr.c,v
retrieving revision 1.40
diff -u -r1.40 file-pr.c
--- file-pr.c   2001/07/15 17:32:31     1.40
+++ file-pr.c   2001/07/23 07:03:06
@@ -729,6 +729,7 @@
 {
   char line [1024]; /* This size doesn't matter here */
   char *buf;
+  size_t buf_size, len;
   const char *from, *to, *subject, *date, *cc;
   PR *pr;
 
@@ -761,17 +762,24 @@
 \n",
            from, to, cc, subject, date);
 
-  while ((fgets (line, sizeof (buf) - 1, infile)) != NULL)
+  /* copy the message from infile, indenting each line by one character
*/
+  len = buf_size = strlen (buf);
+  while ((fgets (line, sizeof (line) - 1, infile)) != NULL)
     {
-      size_t len = strlen (buf);
       size_t lineLen = strlen (line);
 
-      buf = xrealloc (buf, len + lineLen + 2);
+      if (buf_size < len + lineLen + 2)
+       {
+         buf_size += sizeof(line) << 2;
+         buf = xrealloc (buf, buf_size);
+       }
+
       if (buf[len - 1] == '\n')
        {
          buf[len++] = ' ';
        }
       memcpy (buf + len, line, lineLen + 1);
+      len += lineLen;
     }
 
   fclose (infile);

--------------
two fixes:

1) read sizeof(line)-1 chars from the message (<=1024 chars), instead of
sizeof(buf)-1 chars (2 chars).

2) add code to keep track of buf's length in a variable, instead of
doing a strlen(buf) for every loop iteration.

the first fix gives a 20-30x improvement (from hours to minutes) for a
1.4 MB testcase.  the second fix gave a 50x improvement (five minutes to
>10 seconds) for the same testcase.

and some cleanup:

xrealloc()ing buf in 4K chunks only when needed, instead of every loop
iteration.  doesn't seem to change speed, but it feels better...

--
Dirk Bergstrom               dirk@juniper.net
_____________________________________________
Juniper Networks Inc.,          Computer Geek
Tel: 707.433.0564           Fax: 707.433.0769

Gnatsweb help

[S Ramesh]

I have installed gnats 3.113.1 and gnatsweb 2.8.2 in a server. now i have
done with instaalation.. i could able to send and edit PR from web. but the
problem is how to configure it.

1. i gnatsd.access i have a username and passwd for a user with accessleve
and dbase alias name.  now when a user creates a PR his reporters mail id
field is filled by him and he can give any email id. how to restrict this
stuff. is there any possiblity of having unix username and passwd
associated with gnatsd.access user name and passwd.

2. once logged in thru web from a browser froma machine the last login
remains even if we kill the browser. so once logged in the cookies get
enables. how to make it to ask every time for username and passwd.

I am in very much need of this help. gnatsweb users kindly help me to solve
the above mentioned problems.

TIA
Ramesh

Re: Gnatsweb help

[Yngve Svendsen]

At 12:44 23.07.01 +0530, S Ramesh wrote:
>I have installed gnats 3.113.1 and gnatsweb 2.8.2 in a server. now i have
>done with instaalation.. i could able to send and edit PR from web. but the
>problem is how to configure it.
>
>1. i gnatsd.access i have a username and passwd for a user with accessleve
>and dbase alias name.  now when a user creates a PR his reporters mail id
>field is filled by him and he can give any email id. how to restrict this
>stuff. is there any possiblity of having unix username and passwd
>associated with gnatsd.access user name and passwd.

In gnatsweb.pl, you could remove the field from the form and instead have 
it sent with the form as a hidden value. The modification should be fairly 
straightforward if you know a little bit about HTML forms.

>2. once logged in thru web from a browser froma machine the last login
>remains even if we kill the browser. so once logged in the cookies get
>enables. how to make it to ask every time for username and passwd.

One way would of course be to run an OS where cookies are kept private to 
each user. However, I will probably add a "logout" button to the main page 
shortly. This will clear the current login cookie and leave you at the 
login screen afterwards. I'll just have to reach a conclusion in a debate 
with myself over whether the Login Again button should simply have its name 
changed and behaviour slightly modified, or whether we need an all-new button.

A logout button will definitely appear in Gnatsweb 4, but it should be easy 
to port, so it will quite likely also appear in a new Gnatsweb 2.8.x release.

Yngve Svendsen
Gnatsweb maintainer

Re: fix for PR 219 - large PRs or updates take many minutes to process

[Milan Zamazal]

>>>>> "DB" == Dirk Bergstrom <dirk@juniper.net> writes:

    DB> 1) read sizeof(line)-1 chars from the message (<=1024 chars),
    DB> instead of sizeof(buf)-1 chars (2 chars).

    DB> 2) add code to keep track of buf's length in a variable, instead
    DB> of doing a strlen(buf) for every loop iteration.

    DB> the first fix gives a 20-30x improvement (from hours to minutes)
    DB> for a 1.4 MB testcase.  the second fix gave a 50x improvement
    DB> (five minutes to
    >> 10 seconds) for the same testcase.

    DB> and some cleanup:

    DB> xrealloc()ing buf in 4K chunks only when needed, instead of
    DB> every loop iteration.  doesn't seem to change speed, but it
    DB> feels better...

Thanks, Dirk, this is a very useful work!  I applied the patch with
minor changes.

Regards,

Milan Zamazal

-- 
Wasting somebody else's time strikes me as the height of rudeness.
						      Bill Gates