From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Dirk Bergstrom" <dirk@juniper.net>
To: <gnats-devel@sources.redhat.com>
Subject: fix for PR 219 - large PRs or updates take many minutes to process
Date: Mon, 23 Jul 2001 00:11:00 -0000
Message-id: <C0D6C1C24CDBE1449BFEF1B72AFBF3A709A5AD3E@postal.jnpr.net>
X-SW-Source: 2001-q3/msg00068.html

after i submitted PR 219, it bit us again, locking up the server for a
couple hours.  i decided i'd apply my meager C skills, and see if i
couldn't fix it.  it seemed clear it was a badly written loop, and it
wasn't too hard to find it.  turns out there were two problems, a typo
and a needlessly repeated strlen() call.  the patch gives a 1000x
improvement in processing time for large PR updates.

the PR is here:

http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&pr=219&database=g
nats

and the patch is here (untouched by meddlesome MTAs):

http://otisbean.com/file-pr_patch.txt

and here it is for your reading pleasure:

Index: file-pr.c
===================================================================
RCS file: /cvs/gnats/gnats/gnats/file-pr.c,v
retrieving revision 1.40
diff -u -r1.40 file-pr.c
--- file-pr.c   2001/07/15 17:32:31     1.40
+++ file-pr.c   2001/07/23 07:03:06
@@ -729,6 +729,7 @@
 {
   char line [1024]; /* This size doesn't matter here */
   char *buf;
+  size_t buf_size, len;
   const char *from, *to, *subject, *date, *cc;
   PR *pr;
 
@@ -761,17 +762,24 @@
 \n",
            from, to, cc, subject, date);
 
-  while ((fgets (line, sizeof (buf) - 1, infile)) != NULL)
+  /* copy the message from infile, indenting each line by one character
*/
+  len = buf_size = strlen (buf);
+  while ((fgets (line, sizeof (line) - 1, infile)) != NULL)
     {
-      size_t len = strlen (buf);
       size_t lineLen = strlen (line);
 
-      buf = xrealloc (buf, len + lineLen + 2);
+      if (buf_size < len + lineLen + 2)
+       {
+         buf_size += sizeof(line) << 2;
+         buf = xrealloc (buf, buf_size);
+       }
+
       if (buf[len - 1] == '\n')
        {
          buf[len++] = ' ';
        }
       memcpy (buf + len, line, lineLen + 1);
+      len += lineLen;
     }
 
   fclose (infile);

--------------
two fixes:

1) read sizeof(line)-1 chars from the message (<=1024 chars), instead of
sizeof(buf)-1 chars (2 chars).

2) add code to keep track of buf's length in a variable, instead of
doing a strlen(buf) for every loop iteration.

the first fix gives a 20-30x improvement (from hours to minutes) for a
1.4 MB testcase.  the second fix gave a 50x improvement (five minutes to
>10 seconds) for the same testcase.

and some cleanup:

xrealloc()ing buf in 4K chunks only when needed, instead of every loop
iteration.  doesn't seem to change speed, but it feels better...

--
Dirk Bergstrom               dirk@juniper.net
_____________________________________________
Juniper Networks Inc.,          Computer Geek
Tel: 707.433.0564           Fax: 707.433.0769