From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 6788 invoked by alias); 16 Sep 2019 15:06:57 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 6780 invoked by uid 89); 16 Sep 2019 15:06:57 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-5.7 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 spammy=thirdparty, third-party, connect, unofficial X-HELO: rock.gnat.com Received: from rock.gnat.com (HELO rock.gnat.com) (205.232.38.15) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 16 Sep 2019 15:06:54 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by filtered-rock.gnat.com (Postfix) with ESMTP id E44B5117B7E; Mon, 16 Sep 2019 11:06:52 -0400 (EDT) Received: from rock.gnat.com ([127.0.0.1]) by localhost (rock.gnat.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id wOdlpgW6G-mJ; Mon, 16 Sep 2019 11:06:52 -0400 (EDT) Received: from joel.gnat.com (localhost.localdomain [127.0.0.1]) by rock.gnat.com (Postfix) with ESMTP id ABC51117B75; Mon, 16 Sep 2019 11:06:52 -0400 (EDT) Received: by joel.gnat.com (Postfix, from userid 1000) id B73078215B; Mon, 16 Sep 2019 08:06:50 -0700 (PDT) Date: Mon, 16 Sep 2019 15:06:00 -0000 From: Joel Brobecker To: Jason Merrill Cc: "Joseph S. Myers" , Maxim Kuvyrkov , gcc Mailing List , Gerald Pfeifer , Daniel Berlin Subject: Re: GCC Git hooks Message-ID: <20190916150650.GB4945@adacore.com> References: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="/9DWx/yDrRhgMJTb" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-SW-Source: 2019-09/txt/msg00108.txt.bz2 --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-length: 3634 Hello everyone, On Sat, Sep 14, 2019 at 04:53:17PM -0400, Jason Merrill wrote: > At Cauldron this weekend Joel offered to adjust his git hooks > (https://github.com/brobecke/git-hooks), which are already used by gdb > and glibc, to meet GCC's needs. Separately, Joseph volunteered to > deal with converting the gcc-www repository to git and dealing with > those hooks. > > I expect that Joel's hooks should work fine for gcc-www with minimal > changes, but that the main GCC repository will need more work for > bugzilla integration. > > The GCC SVN hooks initially look like pretty vanilla SVN hooks using > svnmailer (http://opensource.perlig.de/svnmailer/); the customized > part of the svnmailer.conf is just > > [libstdcxx] > for_paths = .*(libstdc..-v3)/.* > to_addr = libstdc++-cvs@gcc.gnu.org > > [java] > for_paths = .*(boehm-gc|fastjar|gcjx|gcc/java|libffi|libjava|zlib)/.* > to_addr = java-cvs@gcc.gnu.org > > [gccdefault] > to_addr = gcc-cvs@gcc.gnu.org > bugzilla_to_addr = gcc-bugzilla@gcc.gnu.org > > Pretty short...but the last line relies on Daniel's custom > bugzilla/svnmailer integration (attached below), and it looks like > Joel's hooks don't have anything comparable. Any thoughts about > adjusting Daniel's bugzilla.py vs. pulling in something like > http://www.theoldmonk.net/gitzilla/ ? Looking at the configuration file, I believe the git-hooks should have most, if not all, of the features that would be needed for the GCC repository. In particular, there is already a way to relay commits to third-party tools via calling of a script; GDB uses that to interface with their Bugzilla database. But before I say more I should point to the documentation which, for historical reasons, is available on the GDB wiki rather than the git-hooks GitHub repository. I will fix that, but in the meantime: https://sourceware.org/gdb/wiki/GitHooksUsersGuide I'm attaching a file called project.config, which shows the current configuration for the GDB repository, as it is might be a good starting point for GCC's configuration. Of interest: * We can see that "hooks.mailinglist" is pointing to a script. The purpose of that script is to determine, based on which files changed, which mailinglists should the commit email be sent to. * There is a hooks.style_checker line. Currently, GDB points to a script which does nothing. If GCC has some scripts they want to be run to validate the contents of a file, this is where this can be done. There is no mechanism, really, to say "don't run the style_checker", but it's easy to just pass a null style_checker as done by GDB. * For bugzilla integration, GDB uses the hooks.file-commit-cmd file. I'm attaching the email-to-bugzilla script, although I don't know how useful it will be for GCC. It predates the git-hooks and I just re-used it as is when we switched over to the git-hooks. Given that, it seems like the git-hooks might be ready to support all the needs of the GCC repository? We would need to: - write a script that determines the list of recipients based on the list of files being changed; that should be a trivial adaptation of the script used on the GDB side; - Adapt the script filing the commit with bugzilla - create a refs/meta/config "branch", and add the project.config file with the git-hooks configuration. I can definitely help with the configuration setup phase in whatever capacity you'd like. I would recommend people take a look at the list of options currently available to see what kind of initial configuration we might want to start with. -- Joel --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="project.config" Content-length: 1762 [hooks] from-domain = sourceware.org mailinglist = /git/binutils-gdb.git/hooks-bin/email_to.py # We do not want to force a maximum line length in commit # revision logs, as they get in the way of copy-pasting # debugging session, error messages, logs, etc. max-rh-line-length = 0 # Reject merge commits on a certain number of branches: # - on master: We request that people rebase their changes # before pushing instead (merge commits tend to confuse # git newcommers). # - on GDB release branches: There is a high risk that a merge # commit is a merge from master into the branch, which would # bring a lot more than what the user probably meant to push. # Request that the user cherry-pick his changes. reject-merge-commits = refs/heads/master,refs/heads/gdb-.* # The style checker, applied to the contents of each file being # modified. style-checker = /git/binutils-gdb.git/hooks-bin/style_checker # The URL where we can inspect the commit, inserted in the commit # notification email, and also copy sent to the file-commit-cmd. commit-url = "https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=%(rev)s" # Do not send emails for the following branches (unofficial # third-party branches). no-emails = refs/heads/users/.* # Send a copy to bugzilla if a commit has a PR number in it. file-commit-cmd = "/sourceware/infra/bin/email-to-bugzilla -G 'gdb binutils'" # The script that calls the irker (IRC notification of new # commits). post-receive-hook = /git/binutils-gdb.git/hooks-bin/post-receive --/9DWx/yDrRhgMJTb Content-Type: text/x-python; charset=us-ascii Content-Disposition: attachment; filename="email_to.py" Content-length: 1249 #! /usr/bin/env python import sys ML_MAP = {'bfd': 'bfd-cvs@sourceware.org', 'gdb': 'gdb-cvs@sourceware.org', } OWNER_MAP = ( # BFD file... ('bfd/', 'bfd'), ('binutils/', 'bfd'), ('opcode/', 'bfd'), ('cpu/', 'bfd'), ('elfcpp/', 'bfd'), ('gas/', 'bfd'), ('gold/', 'bfd'), ('gprof/', 'bfd'), ('include/', 'bfd'), ('ld/', 'bfd'), ('opcodes/', 'bfd'), # GDB files... ('gdb/', 'gdb'), ('readline/', 'gdb'), ('sim/', 'gdb'), ) EVERYONE = set(ML_MAP[ml_key] for ml_key in ML_MAP) def ml_from_filename(filename): for (path, ml_key) in OWNER_MAP: if filename.startswith(path): return ML_MAP[ml_key] # Not found in map, it is a common file. return EVERYONE result = set() for filename in sys.stdin: ml = ml_from_filename(filename) if isinstance(ml, basestring): result.add(ml) else: result.update(ml) if len(result) >= len(EVERYONE): # We have iterated over enough entries to know already # that we have selected all possible recipients. So # stop now. break if not result: # No files given, return EVERYONE result = EVERYONE print '\n'.join(sorted(result)) --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=email-to-bugzilla Content-length: 3586 #!/usr/bin/perl # -*-Perl-*- # # Perl filter to handle the log messages from the checkin of files in # a directory. This script will group the lists of files by log # message, and mail a single consolidated log message at the end of # the commit. # # This file assumes a pre-commit checking program that leaves the # names of the first and last commit directories in a temporary file. # # Contributed by David Hampton # # hacked greatly by Greg A. Woods # # Then chopped down just to send bugzilla email, for git. use POSIX; use DBI; # # Configurable options # $TMPDIR = "/sourceware/cvs-tmp"; $BMAILER = "/usr/sbin/sendmail"; # # Subroutines # sub see_if_bugzilla_bug_exists { local ($dbh, $product, $id) = @_; # Split $PRODUCT and SQL-ify. my $sql_product = ''; foreach $i (split (/\s+/, $product)) { if ($sql_product ne '') { $sql_product .= ', '; } $sql_product .= "'" . $i . "'"; } my $sth2 = $dbh->prepare ("SELECT COUNT(*) from bugs where bug_id = $id and product_id = any (select products.id from products where name in ($sql_product))") or return 0; $sth2->execute() or return 0; my $count = $sth2->fetchrow_array (); return $count > 0; } sub mail_bug_notification { local($name, $subject, @text) = @_; open(MAIL, "| $BMAILER -f\"cvs-commit\@gcc.gnu.org\" $name"); print MAIL "From: cvs-commit\@gcc.gnu.org\n"; print MAIL "Subject: $subject\n"; print MAIL "To: $name\n"; print MAIL "Content-Type: text/plain; charset=UTF-8\n"; print MAIL "\n"; print MAIL join("\n", @text), "\n"; close(MAIL); } # # Main Body # # Initialize basic variables # $debug = 0; chop($hostname = `hostname`); # Parse command line arguments. while (@ARGV) { $arg = shift @ARGV; if ($arg eq '-d') { $debug = 1; print STDERR "Debug turned on...\n"; } elsif ($arg eq '-G') { ($bugzillaproduct) && die("Too many '-G' args\n"); $bugzillaproduct = shift @ARGV; } } if ($hostname !~ /\./) { chop($domainname = `domainname`); $hostdomain = $hostname . "." . $domainname; } else { $hostdomain = $hostname; } # Used with sprintf to form name of Gnats notification mailing list. # %s argument comse from -G option. $GNATS_MAIL_FORMAT = "%s-bugzilla\@$hostdomain"; # Collect the body of the commit message. binmode STDIN, ":utf8"; while () { chop; push (@text, $_); } $log_txt = join ("\n", @text); %done_ids = {}; while ($log_txt =~ m/[^Aa](?:bug|PR|BZ)\s+\#?\s*(?:[a-z+-]+\/)?(?:\/)?(\d+)(.*)$/si) { $bug_id = $1; $log_txt = $2; if (!defined $done_ids{$bug_id}) { $done_ids{$bug_id} = 1; # Send mail to Bugzilla, if required. if ($bugzillaproduct ne '') { my $dbh = undef; if ($bugzillaproduct eq 'gcc') { $dbh = DBI->connect ("dbi:mysql:bugs", "swbugz", "everythingroses"); } else # elsif ($bugzillaproduct eq 'glibc') { $dbh = DBI->connect ("dbi:mysql:sourcesbugs", "swbugz", "everythingroses"); } if ($debug) { print STDERR "Attempting to see if bug $bug_id exists\n"; } if (defined $dbh && &see_if_bugzilla_bug_exists ($dbh, $bugzillaproduct, $bug_id)) { if ($debug) { print STDERR "It does\n"; } if ($bugzillaproduct ne 'gcc') { &mail_bug_notification( sprintf ($GNATS_MAIL_FORMAT, "sourceware"), "[Bug $bug_id]", @text); } else { &mail_bug_notification( sprintf ($GNATS_MAIL_FORMAT, $bugzillaproduct), "[Bug $bug_id]", @text); } } if (defined $dbh) { $dbh->disconnect; } } } } exit 0; --/9DWx/yDrRhgMJTb--