patch to log_accum

patch to log_accum

[Ben Elliston]

A patch to /usr/sourceware/bin/log_accum appears below that adds a
`-i' option to emit SQL INSERT statements to files in a spool
directory.  I have proofread it a number of times and believe it is
innocuous.  It certainly does nothing unless `-i' is selectively
passed by each project's loginfo file.

The reason behind this patch is that, for each commit, a file in
/var/spool/cvsql (or some other location) is created containing SQL
statements that a cron job of mine will process periodically.  The CVS
commit information will be placed in an SQL database.  I am currently
working on a web-based front-end (called `CVSQL') to allow all sorts
of expressive queries to be issued.

I would really like to see this patch go in, and so am willing to work
to make it palatable to the sources overseers. ;-)

Cheers, Ben


--- log_accum~	Fri Oct 25 10:51:59 2002
+++ log_accum	Fri Oct 25 10:52:57 2002
@@ -27,6 +27,7 @@
 #       -D DOMAIN       - Domain from which mail should appear
 #       -l              - Update list of modified files in CVSROOT/commits.  
 #                         Must be run using %{sVv}.
+#       -i              - Generate SQL INSERT statements for CVSQL.
 
 use POSIX;
 
@@ -35,6 +36,7 @@
 #
 
 $TMPDIR = "/sourceware/cvs-tmp";
+$SQL_SPOOLDIR = "/var/spool/cvsql";
 
 # Set this to something that takes "-s"
 $MAILER	       = "/usr/bin/Mail";
@@ -311,6 +313,45 @@
     close(FILE);
 }
 
+# Return a sequence of SQL inserts.
+sub generate_cvsql_inserts {
+    local ($dir, $branch, @files) = @_;
+    local (@sp, @result);
+    local ($now) = strftime ("%Y/%m/%d %H:%M:%S", localtime);
+    local ($msg) = join ("\n", @log_lines);
+
+    if ($branch eq '') {
+	$branch = 'null';
+    } else {
+	$branch = '\'' . $branch . '\'';
+    }
+
+    # Escape certain characters for psql(1).
+    $msg =~ s,(\'|\"|\\),\\$1,g;
+
+    push (@result, '\\set id `psql -d cvs -A -t -c "select nextval (\'changesetid_seq\');"`');
+    push (@result, 'INSERT INTO changeset (id,author,date,msg,cvsroot) values (:id,\'' .
+	  $login . '\',\'' . $now . '\',\'' . $msg . '\',\'' . $hostdomain . ':' . $cvsroot . '\');');
+
+    foreach (@files) {
+	# List is (FILE OLD-REV NEW-REV).
+	@sp = split (',');
+	if ($sp[1] eq 'NONE') {
+	    $sp[1] = 'null';
+	} else {
+	    $sp[1] = '\'' . $sp[1] . '\'';
+	}
+	if ($sp[2] eq 'NONE') {
+	    $sp[2] = 'null';
+	} else {
+	    $sp[2] = '\'' . $sp[2] . '\'';
+	}
+	push (@result, 'INSERT INTO change (filename,prerev,postrev,branch,changeset) values (\'' .
+	      $dir . '/' . $sp[0] . '\',' . $sp[1] . ',' . $sp[2] . ',' . $branch . ',:id);');
+    }
+    return @result;
+}
+
 # Return a list of all cvsweb URLs for this commit.
 sub generate_cvsweb_urls {
     local ($dir, $branch, @files) = @_;
@@ -564,6 +605,7 @@
 $do_cvsweb = 0;
 $cvsweb_name = '';
 $do_modlist = 0;
+$do_cvsql = 0;
 
 # parse command line arguments (file list is seen as one arg)
 #
@@ -596,6 +638,8 @@
 	$hostdomain = shift @ARGV;
     } elsif ($arg eq '-l') {
 	$do_modlist = 1;
+    } elsif ($arg eq '-i') {
+	$do_cvsql = 1;
     } else {
 	($donefiles) && die("Too many arguments!  Check usage.\n");
 	$donefiles = 1;
@@ -762,6 +806,11 @@
 				   @files[1 .. $#files]);
 }
 
+if ($do_cvsql) {
+    @sql_stmts = &generate_cvsql_inserts (join ('/', @path), $branch,
+					  @files[1 .. $#files]);
+}
+
 if ($do_modlist) {
     @modlist = generate_modlist (join ('/', @path),
                               \@changed_files, \@added_files, \@removed_files,
@@ -923,18 +972,42 @@
 #
 &mail_notification($mailto, $subject_txt, @text);
 
-# Send mail to Gnats, if required.
-if ($gnatsdb ne '') {
-    $log_txt = join ("\n", @log_txt);
-    while ($log_txt =~ m,PR ([a-z.+]+/[0-9]+)(.*)$,s) {
-	$pr = $1;
-	$log_txt = $2;
+$log_txt = join ("\n", @log_txt);
+while ($log_txt =~ m,PR ([a-z.+]+/[0-9]+)(.*)$,s) {
+    $pr = $1;
+    $log_txt = $2;
+
+    # Send mail to Gnats, if required.
+    if ($gnatsdb ne '') {
 	$file = sprintf ($GNATS_ROOT_FORMAT, $gnatsdb) . "/" . $pr;
 	if (-f $file) {
 	    &mail_notification(sprintf ($GNATS_MAIL_FORMAT, $gnatsdb),
 			       $pr, @text);
 	}
     }
+    if ($do_cvsql) {
+	push(@sql_stmts, 'INSERT INTO bugreport (changeset,id) values (:id, \'' . $pr . '\');');
+    }
+}
+
+# Write out an SQL batch file.
+#
+if ($do_cvsql) {
+    $extension = $$;
+    do
+    {
+        $extension++;
+        $spoolfile = $SQL_SPOOLDIR . "/tmp." . $extension;
+        print "trying " . $spoolfile . "\n";
+    } while (!sysopen SQL, $spoolfile, O_WRONLY | O_CREAT | O_EXCL, 0644);
+
+    print SQL "BEGIN;";
+    if ($debug) { print join("\n", @sql_stmts), "\n"; }
+    print SQL join("\n", @sql_stmts), "\n";
+    print SQL "COMMIT;\n";
+    close (SQL);
+    $newfile = $SQL_SPOOLDIR . "/sql." . $extension;
+    rename $spoolfile, $newfile
 }
 
 # cleanup

Re: patch to log_accum

[Jason Molenda]

Hi Ben, 

I just looked it over very briefly without thinking, but didn't
see anything obviously problematic.  Please check out the testcvs
repository and do a little testing there.  If everything is
reasonable, it's OK to check it into the main log_accum.  When you
do, send a note to overseers letting everyone know that any newly
reported cvs commit problems should be brought to your attention.  :-)

Unless someone has a differing opinion on the matter, which I doubt.
With things like log_accum, as long as you don't screw over anyone,
or are able to clean up after yourself if you do, it's pretty open
to making changes.

I've given you write access to both the testcvs and sourceware
repositories.

BTW you might want to look at bonsai - http://bonsai.mozilla.org/
It's one of the original mozilla infrastructure bits, along with
tinderbox and bugzilla.  It's what you're thinking of making, but
a few years farther along. :-)  I haven't looked at how it's implemented,
but my small experience with the moz infra tools is that they work
best on mozilla.org, and you're often on your own to make them work
elsewhere.

Jason

PS- As far as log_accum itself goes, I think the best branch of
development is the one used in the FreeBSD repository.  They don't
have the same feature set that we have, but they've done some
pretty reasonable code cleanup on this hack of a script.

Re: patch to log_accum

[Ben Elliston]

Hi Crash,

>>>>> "Jason" == Jason Molenda <jason-swarelist@molenda.com> writes:

  Jason> I just looked it over very briefly without thinking, but
  Jason> didn't see anything obviously problematic.  Please check out
  Jason> the testcvs repository and do a little testing there.  If
  Jason> everything is reasonable, it's OK to check it into the main
  Jason> log_accum.  When you do, send a note to overseers letting
  Jason> everyone know that any newly reported cvs commit problems
  Jason> should be brought to your attention.  :-)

I did some testing using testcvs (my, that's very handy).  Anyway, at
your request, here is such a note: overseers, I just changed
log_accum.  If it breaks, blame me!

There should be no visible changes for any project other than SID,
which I am using as the basis for some cautious and slow testing.

Ben

Re: patch to log_accum

[Ben Elliston]

I have tested log_accum over the weekend and am satisfied that it is
working well.  I had a couple of refinements to make, but mostly they
were cosmetic.

Now, whose approval do I need to commit changes to, say, /cvs/gcc's
CVSROOT/loginfo?  Overseers or a GCC maintainer?

Cheers, Ben

Re: patch to log_accum

[Jason Molenda]

On Sun, Oct 27, 2002 at 07:06:36PM +1100, Ben Elliston wrote:

> 
> Now, whose approval do I need to commit changes to, say, /cvs/gcc's
> CVSROOT/loginfo?  Overseers or a GCC maintainer?
> 

gcc is using the same script.  You mean you want to enable the SQL
logging stuff?

The gcc maintainers generally don't care about infrastructure
changing under them as long as it doesn't break anything or change
behaviors.  I normally do these sorts of changes without telling
anyone--much like the sourceware side, really. :-)  But in my own
mind, I'm always re-examining everything I'm doing for possible
side effects or bugs before enabling it on the gcc side -- not that
I'm not careful with the sourceware side, but that feels like
"our site", whereas the gcc side feels more like "not our site",
so there's an extra burden of carefulness involved.

As long as we never break the gcc side of the site, my policy works
fine.  The first time they get hosed by something changing behind
the scenes, I expect there'll be some discussion about process for
these sorts of things..

Jason

Re: patch to log_accum

[Ben Elliston]

>>>>> "Jason" == Jason Molenda <jason-swarelist@molenda.com> writes:

  Jason> gcc is using the same script.  You mean you want to enable the SQL
  Jason> logging stuff?

Yes.

  Jason> As long as we never break the gcc side of the site, my policy
  Jason> works fine.  The first time they get hosed by something
  Jason> changing behind the scenes, I expect there'll be some
  Jason> discussion about process for these sorts of things..

I am being super careful and have tested extensively in the `testcvs'
area.  If I don't hear any violent objections, I will commit a change
tomorrow to enable the SQL logging for the binutils directories
(binutils, bfd, et al).  If it continues to work correctly, then I
will start advertising it more widely.

Ben