git conversion of GCC wwwdocs repository

git conversion of GCC wwwdocs repository

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 2595 bytes --]

I've done a git conversion of the GCC wwwdocs repository (from CVS, using 
cvs-fast-export), plus written a git post-receive hook intended to 
correspond to what the CVS hook does.  (For the CVS hook, see loginfo in 
the CVSROOT, plus the scripts such as htdocs-checkout referenced from 
there.)

To access the test conversion (given SSH access to gcc.gnu.org), do:

git clone git+ssh://gcc.gnu.org/home/gccadmin/wwwdocs-test.git

(This only needs the usual restricted SSH access used to commit to GCC, 
not shell access.)

Note that the test repository does *not* have the post-receive hook 
enabled; pushing commits to it won't do anything useful; it's simply there 
to make it convenient to review the converted history.  In practice, I 
expect the post-receive hook may need bugs ironed out once in use in 
production, because it's hard to test when not hooked up to the real 
website.

I've attached the conversion scripts and proposed post-receive hook.  
do-download checks out cvs-fast-export and ESR's gcc-conversion repository 
(for the author map) and rsyncs the wwwdocs CVS repository.  do-convert 
builds cvs-fast-export and does the conversion (with a few extra authors 
in the author map, where e.g. new since the last update to the author map 
or only ever committed to wwwdocs not GCC sources; the names for htdigid 
and www are somewhat arbitrary, www commits would actually have been some 
human or humans committing as that user by accident); once cvs-fast-export 
has been built, this takes about 15 seconds.  All non-master tags and 
branches are removed, since they are clearly just accidents from tags / 
branches having been applied to the whole CVS repository rather than just 
to the GCC sources proper.

post-receive is the proposed hook.  It calls post-receive-email to send 
email (could no doubt be adjusted to use AdaCore hooks or any other 
email-sending system), then follows logic intended to replicate the old 
hooks to update a checkout of wwwdocs and call the preprocess script on 
new or modified files.

I'd expect the post-receive script to go in wwwdocs/bin (with the hook 
directory having a symlink to there).  There would of course be website 
changes needed to update the references to CVS, and the preprocess script 
would change to ignore .git when doing global preprocessing on the site.

I don't know whether either the system for updating www.gnu.org's copy of 
the site, or Gerald's HTML validation system, would need updating for a 
move of wwwdocs to git.

Any comments on the conversion or hook?

-- 
Joseph S. Myers
joseph@codesourcery.com

[-- Attachment #2: Type: text/plain, Size: 270 bytes --]

#!/bin/bash

set -e

TOP=$(cd $(dirname "$0") && pwd)

DOWNLOAD=$TOP/download

mkdir "$DOWNLOAD"
cd "$DOWNLOAD"
git clone https://gitlab.com/esr/cvs-fast-export.git
git clone https://gitlab.com/esr/gcc-conversion.git
rsync -a gcc.gnu.org::gcc-cvs/wwwdocs .

[-- Attachment #3: Type: text/plain, Size: 1465 bytes --]

#!/bin/bash

set -e

TOP=$(cd $(dirname "$0") && pwd)

REPO=$TOP/wwwdocs.git
mkdir $REPO
cd $REPO
git init --bare --shared

DOWNLOAD=$TOP/download

cd "$DOWNLOAD/cvs-fast-export"
make -j cvs-fast-export

grep -a = $DOWNLOAD/gcc-conversion/gcc.map | sed -e 's/>.*/>/' > $TOP/gcc.map
# dje specially handled for GCC conversion because of ambiguity with
# gcc2 repository.  dimitar, shorne postdate GCC conversion map; not
# needed once that is updated.  Others not needed in GCC conversion
# map, as only committed to wwwdocs not to main sources repository.
cat >> $TOP/gcc.map <<EOF
dje = David Edelsohn <dje.gcc@gmail.com>
dimitar = Dimitar Dimitrov <dimitar@dinux.eu>
shorne = Stafford Horne <shorne@gmail.com>
angela = Angela Marie Thomas <angela@releasedominatrix.com>
brads = Bradley Schatz <brads@gcc.gnu.org>
gumby = David Henkel-Wallace <gumby@gcc.gnu.org>
kingdon = Jim Kingdon <kingdon@gcc.gnu.org>
htdigid = htdigid automatic commit <htdigid@gcc.gnu.org>
www = www user <www@gcc.gnu.org>
EOF

cd "$DOWNLOAD/wwwdocs"
find . -name '*,v' | sort | ../cvs-fast-export/cvs-fast-export -A $TOP/gcc.map | (cd "$REPO" && git fast-import)

cd "$REPO"
# CVS branches and tags are accidents from whole-CVS-repository
# branching / tagging, remove.
for ref in $(git for-each-ref --format='%(refname)'); do
    if [ "$ref" != "refs/heads/master" ]; then
	git update-ref -d "$ref"
    fi
done
git gc --aggressive --prune=all

[-- Attachment #4: Type: text/plain, Size: 1398 bytes --]

#!/bin/bash

# The post-receive hook receives, on standard input, a series of lines
# of the form:
#
# <old-value> SP <new-value> SP <ref-name> LF
#
# describing ref updates that have just occurred.  In the case of this
# repository, only updates to refs/heads/master are expected, and only
# such updates should result in automatic website updates.

TOP_DIR=/www/gcc

exec >> "$TOP_DIR/updatelog" 2>&1

export QMAILHOST=gcc.gnu.org

tmp=$(mktemp)
cat > "$tmp"

# Send commit emails.  Appropriate config values should be set in the
# git repository for this.
/sourceware/libre/infra/bin/post-receive-email < "$tmp"

# Update web page checkouts, if applicable.
while read old_value new_value ref_name; do
    if [ "$ref_name" != "refs/heads/master" ]; then
	continue
    fi
    unset GIT_DIR
    unset GIT_WORK_TREE
    cd "$TOP_DIR/wwwdocs-checkout"
    git pull --quiet
    # $TOP_DIR/bin, $TOP_DIR/cgi-bin and $TOP_DIR/htdocs-preformatted
    # should be symlinks into wwwdocs-checkout.
    git diff --name-only "$old_value..$new_value" | while read file; do
	case "$file" in
	    (htdocs/*)
		;;
	    (*)
		continue
		;;
	esac
	dir="${file%/*}"
	if ! [ -d "$TOP_DIR/$dir" ]; then
	    mkdir "$TOP_DIR/$dir"
	    chmod 2775 "$TOP_DIR/$dir"
	fi
	if [ -f "$file" ]; then
	    /www/gcc/bin/preprocess "${file#htdocs/}"
	fi
    done
done < "$tmp"

Re: git conversion of GCC wwwdocs repository

[Joseph Myers]

Would anyone like to make any comments on this conversion from CVS to git?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: git conversion of GCC wwwdocs repository

[Jonathan Wakely]

On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
>
> Would anyone like to make any comments on this conversion from CVS to git?

It looks pretty good. I note that the author map just uses the
committer's current email address, meaning I have commits using my
@redhat.com address nearly a decade before I started working at Red
Hat. But that's a small price to pay for moving from CVS to Git in my
opinion. And t's arguably correct to have all my commits under one
identity rather than several different ones anyway.

Re: git conversion of GCC wwwdocs repository

[Joseph Myers]

On Tue, 24 Sep 2019, Jonathan Wakely wrote:

> On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
> >
> > Would anyone like to make any comments on this conversion from CVS to git?
> 
> It looks pretty good. I note that the author map just uses the
> committer's current email address, meaning I have commits using my
> @redhat.com address nearly a decade before I started working at Red
> Hat. But that's a small price to pay for moving from CVS to Git in my
> opinion. And t's arguably correct to have all my commits under one
> identity rather than several different ones anyway.

If people prefer to use @gcc.gnu.org addresses, that's a small matter of 
adjusting the postprocessing of the author map from the gcc-conversion 
repository to replace the addresses there with username@gcc.gnu.org while 
keeping the names from that author map.  (The postprocessing is needed 
anyway for this conversion because cvs-fast-export uses a more restricted 
author map syntax than reposurgeon does.  Because there is no ChangeLog in 
wwwdocs, more sophisticated systems for identifying the relevant email 
address for each commit from the ChangeLog aren't practical the same way 
they are for the main GCC repository.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: git conversion of GCC wwwdocs repository

[Jonathan Wakely]

On Tue, 24 Sep 2019 at 23:15, Joseph Myers wrote:
>
> On Tue, 24 Sep 2019, Jonathan Wakely wrote:
>
> > On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
> > >
> > > Would anyone like to make any comments on this conversion from CVS to git?
> >
> > It looks pretty good. I note that the author map just uses the
> > committer's current email address, meaning I have commits using my
> > @redhat.com address nearly a decade before I started working at Red
> > Hat. But that's a small price to pay for moving from CVS to Git in my
> > opinion. And t's arguably correct to have all my commits under one
> > identity rather than several different ones anyway.
>
> If people prefer to use @gcc.gnu.org addresses, that's a small matter of
> adjusting the postprocessing of the author map from the gcc-conversion
> repository to replace the addresses there with username@gcc.gnu.org while
> keeping the names from that author map.

I don't feel strongly about it.

Re: git conversion of GCC wwwdocs repository

[Jason Merrill]

On Tue, Sep 24, 2019 at 6:16 PM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 24 Sep 2019, Jonathan Wakely wrote:
> > On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
> > >
> > > Would anyone like to make any comments on this conversion from CVS to git?
> >
> > It looks pretty good. I note that the author map just uses the
> > committer's current email address, meaning I have commits using my
> > @redhat.com address nearly a decade before I started working at Red
> > Hat. But that's a small price to pay for moving from CVS to Git in my
> > opinion. And t's arguably correct to have all my commits under one
> > identity rather than several different ones anyway.
>
> If people prefer to use @gcc.gnu.org addresses, that's a small matter of
> adjusting the postprocessing of the author map from the gcc-conversion
> repository to replace the addresses there with username@gcc.gnu.org while
> keeping the names from that author map.  (The postprocessing is needed
> anyway for this conversion because cvs-fast-export uses a more restricted
> author map syntax than reposurgeon does.  Because there is no ChangeLog in
> wwwdocs, more sophisticated systems for identifying the relevant email
> address for each commit from the ChangeLog aren't practical the same way
> they are for the main GCC repository.)

That would be my preference.

Jason

Re: git conversion of GCC wwwdocs repository

[Thomas Schwinge]

Hi!

On 2019-09-24T22:51:18-0400, Jason Merrill <jason@redhat.com> wrote:
> On Tue, Sep 24, 2019 at 6:16 PM Joseph Myers <joseph@codesourcery.com> wrote:
>> On Tue, 24 Sep 2019, Jonathan Wakely wrote:
>> > On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
>> > > Would anyone like to make any comments on this conversion from CVS to git?

Again, thanks for working on this!  The conversion seems good to me (but
I only had a quick look).


As mentioned at the Cauldron, I shall uninstall CVS from my systems once
this is done.  ;-)


>> > I note that the author map just uses the
>> > committer's current email address, meaning I have commits using my
>> > @redhat.com address nearly a decade before I started working at Red
>> > Hat. But that's a small price to pay for moving from CVS to Git in my
>> > opinion. And t's arguably correct to have all my commits under one
>> > identity rather than several different ones anyway.

(That last sentence/point I don't understand.)

>> If people prefer to use @gcc.gnu.org addresses, that's a small matter of
>> adjusting the postprocessing of the author map from the gcc-conversion
>> repository to replace the addresses there with username@gcc.gnu.org while
>> keeping the names from that author map.  (The postprocessing is needed
>> anyway for this conversion because cvs-fast-export uses a more restricted
>> author map syntax than reposurgeon does.  Because there is no ChangeLog in
>> wwwdocs, more sophisticated systems for identifying the relevant email
>> address for each commit from the ChangeLog aren't practical the same way
>> they are for the main GCC repository.)
>
> That would be my preference.

It's also my preference to just use '@gcc.gnu.org' addresses, with or
even without user names mapped/prepended.

Or even use an "unknown" Git author for all Git commits, and use each
revision's CVS author for the Git committer.  After all, we can't be sure
that the CVS author is the actual author (and thus Git author) of the
respective CVS revision.

(That'd also have been my pragmatic suggestion for the main GCC
reporitory conversion, but I see that people now have invested effort in
a more sophisticated solution.)


Grüße
 Thomas

Re: git conversion of GCC wwwdocs repository

[Jonathan Wakely]

On Wed, 25 Sep 2019 at 11:08, Thomas Schwinge <thomas@codesourcery.com> wrote:
>
> Hi!
>
> On 2019-09-24T22:51:18-0400, Jason Merrill <jason@redhat.com> wrote:
> > On Tue, Sep 24, 2019 at 6:16 PM Joseph Myers <joseph@codesourcery.com> wrote:
> >> On Tue, 24 Sep 2019, Jonathan Wakely wrote:
> >> > On Tue, 24 Sep 2019 at 21:46, Joseph Myers wrote:
> >> > > Would anyone like to make any comments on this conversion from CVS to git?
>
> Again, thanks for working on this!  The conversion seems good to me (but
> I only had a quick look).
>
>
> As mentioned at the Cauldron, I shall uninstall CVS from my systems once
> this is done.  ;-)

Me too! :-)

> >> > I note that the author map just uses the
> >> > committer's current email address, meaning I have commits using my
> >> > @redhat.com address nearly a decade before I started working at Red
> >> > Hat. But that's a small price to pay for moving from CVS to Git in my
> >> > opinion. And t's arguably correct to have all my commits under one
> >> > identity rather than several different ones anyway.
>
> (That last sentence/point I don't understand.)

I mean that there's not much value in having my past commits listed as
coming from various "different" authors:

Jonathan Wakely <xxx at compsoc.man.ac.uk>
Jonathan Wakely <xxx at kayari.org>
Jonathan Wakely <xxx at gmail.com>
Jonathan Wakely <xxx at gcc.gnu.org>
Jonathan Wakely <xxx at redhat.com>

All of those "identities" are the same person, so making all my
commits use the same Author is arguably correct, even if I was using
different addresses at different times.

Using the @gcc.gnu.org address for all my past commits might be the
least worst option, because it's "me" and that address was valid in
2004 and is still valid now, unlike the other options.

Re: git conversion of GCC wwwdocs repository

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

A version of the conversion using @gcc.gnu.org addresses is now available:

git clone git+ssh://gcc.gnu.org/home/gccadmin/wwwdocs-test2.git

My current versions of the conversion scripts / proposed hook are 
attached.  The rest of the description of the first test conversion still 
applies.

-- 
Joseph S. Myers
jsm@polyomino.org.uk

[-- Attachment #2: Type: text/plain, Size: 1527 bytes --]

#!/bin/bash

set -e

TOP=$(cd $(dirname "$0") && pwd)

REPO=$TOP/wwwdocs.git
mkdir $REPO
cd $REPO
git init --bare --shared

DOWNLOAD=$TOP/download

cd "$DOWNLOAD/cvs-fast-export"
make -j cvs-fast-export

grep -a = $DOWNLOAD/gcc-conversion/gcc.map | sed -e 's/^\([^=]*\) = \([^<]*\) <.*/\1 = \2 <\1@gcc.gnu.org>/' > $TOP/gcc.map
# dje specially handled for GCC conversion because of ambiguity with
# gcc2 repository.  dimitar, shorne postdate GCC conversion map; not
# needed once that is updated.  Others not needed in GCC conversion
# map, as only committed to wwwdocs not to main sources repository.
cat >> $TOP/gcc.map <<EOF
dje = David Edelsohn <dje@gcc.gnu.org>
dimitar = Dimitar Dimitrov <dimitar@gcc.gnu.org>
shorne = Stafford Horne <shorne@gcc.gnu.org>
angela = Angela Marie Thomas <angela@gcc.gnu.org>
brads = Bradley Schatz <brads@gcc.gnu.org>
gumby = David Henkel-Wallace <gumby@gcc.gnu.org>
kingdon = Jim Kingdon <kingdon@gcc.gnu.org>
htdigid = htdigid automatic commit <htdigid@gcc.gnu.org>
www = www user <www@gcc.gnu.org>
EOF

cd "$DOWNLOAD/wwwdocs"
find . -name '*,v' | sort | ../cvs-fast-export/cvs-fast-export -A $TOP/gcc.map | (cd "$REPO" && git fast-import)

cd "$REPO"
# CVS branches and tags are accidents from whole-CVS-repository
# branching / tagging, remove.
for ref in $(git for-each-ref --format='%(refname)'); do
    if [ "$ref" != "refs/heads/master" ]; then
	echo "Removing $ref"
	git update-ref -d "$ref"
    fi
done
git gc --aggressive --prune=all

[-- Attachment #3: Type: text/plain, Size: 270 bytes --]

#!/bin/bash

set -e

TOP=$(cd $(dirname "$0") && pwd)

DOWNLOAD=$TOP/download

mkdir "$DOWNLOAD"
cd "$DOWNLOAD"
git clone https://gitlab.com/esr/cvs-fast-export.git
git clone https://gitlab.com/esr/gcc-conversion.git
rsync -a gcc.gnu.org::gcc-cvs/wwwdocs .

[-- Attachment #4: Type: text/plain, Size: 1411 bytes --]

#!/bin/bash

# The post-receive hook receives, on standard input, a series of lines
# of the form:
#
# <old-value> SP <new-value> SP <ref-name> LF
#
# describing ref updates that have just occurred.  In the case of this
# repository, only updates to refs/heads/master are expected, and only
# such updates should result in automatic website updates.

TOP_DIR=/www/gcc

exec >> "$TOP_DIR/updatelog" 2>&1

export QMAILHOST=gcc.gnu.org

tmp=$(mktemp)
cat > "$tmp"

# Send commit emails.  Appropriate config values should be set in the
# git repository for this.
/sourceware/libre/infra/bin/post-receive-email < "$tmp"

# Update web page checkouts, if applicable.
while read old_value new_value ref_name; do
    if [ "$ref_name" != "refs/heads/master" ]; then
	continue
    fi
    unset GIT_DIR
    unset GIT_WORK_TREE
    cd "$TOP_DIR/wwwdocs-checkout"
    git pull --quiet
    # $TOP_DIR/bin, $TOP_DIR/cgi-bin and $TOP_DIR/htdocs-preformatted
    # should be symlinks into wwwdocs-checkout.
    git diff --name-only "$old_value..$new_value" | while read file; do
	case "$file" in
	    (htdocs/*)
		;;
	    (*)
		continue
		;;
	esac
	dir="${file%/*}"
	if ! [ -d "$TOP_DIR/$dir" ]; then
	    mkdir "$TOP_DIR/$dir"
	    chmod 2775 "$TOP_DIR/$dir"
	fi
	if [ -f "$file" ]; then
	    /www/gcc/bin/preprocess "${file#htdocs/}"
	fi
    done
done < "$tmp"

rm "$tmp"

Re: git conversion of GCC wwwdocs repository

[Segher Boessenkool]

On Wed, Sep 25, 2019 at 12:02:42PM +0100, Jonathan Wakely wrote:
> I mean that there's not much value in having my past commits listed as
> coming from various "different" authors:
> 
> Jonathan Wakely <xxx at compsoc.man.ac.uk>
> Jonathan Wakely <xxx at kayari.org>
> Jonathan Wakely <xxx at gmail.com>
> Jonathan Wakely <xxx at gcc.gnu.org>
> Jonathan Wakely <xxx at redhat.com>
> 
> All of those "identities" are the same person, so making all my
> commits use the same Author is arguably correct, even if I was using
> different addresses at different times.
> 
> Using the @gcc.gnu.org address for all my past commits might be the
> least worst option, because it's "me" and that address was valid in
> 2004 and is still valid now, unlike the other options.

And using a different address is *incorrect*, and it is quite nasty if
you would be associating a commit with the wrong employer.  @gcc.gnu.org
is fine of course.


Segher

Re: git conversion of GCC wwwdocs repository

[Segher Boessenkool]

On Wed, Sep 25, 2019 at 04:49:58PM +0000, Joseph Myers wrote:
> A version of the conversion using @gcc.gnu.org addresses is now available:
> 
> git clone git+ssh://gcc.gnu.org/home/gccadmin/wwwdocs-test2.git

This looks great, thanks!

Can we test committing to this repo as well?  Are hooks set up already?
Am I too impatient?


Segher

Re: git conversion of GCC wwwdocs repository

[Joseph Myers]

On Wed, 25 Sep 2019, Segher Boessenkool wrote:

> On Wed, Sep 25, 2019 at 04:49:58PM +0000, Joseph Myers wrote:
> > A version of the conversion using @gcc.gnu.org addresses is now available:
> > 
> > git clone git+ssh://gcc.gnu.org/home/gccadmin/wwwdocs-test2.git
> 
> This looks great, thanks!
> 
> Can we test committing to this repo as well?  Are hooks set up already?

The hooks aren't set up.  Setting up the hooks will involve moving various 
directories around in /www/gcc on sourceware to replace CVS checkouts with 
symlinks into a git checkout, i.e. it's part of the actual move from CVS 
to git, since the main thing the hooks do is run the preprocessor to 
update the live website.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: git conversion of GCC wwwdocs repository

[Joseph Myers]

As far as I can tell from the discussion, people are happy with the 
version of the conversion using @gcc.gnu.org addresses.  When shall we (do 
a final conversion if there are any commits postdating that one and) 
switch over to it as the live repository, set up hooks and update 
documentation of editing the website?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: git conversion of GCC wwwdocs repository

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 758 bytes --]

Hi Joseph!

On 2019-09-30T19:56:39+0000, Joseph Myers <joseph@codesourcery.com> wrote:
> As far as I can tell from the discussion, people are happy with the 
> version of the conversion using @gcc.gnu.org addresses.  When shall we (do 
> a final conversion if there are any commits postdating that one and) 
> switch over to it as the live repository, set up hooks and update 
> documentation of editing the website?

Given that nobody has raised any adverse opinions: can do that anytime
now?

Doing 'wwwdocs' before the sources repository conversion, we can use it
as a testbed for hooks setup: commit/push hook obviously, but also
Bugzilla integration, and fast-forward-only requirement for the 'master'
branch, etc.


Grüße
 Thomas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: git conversion of GCC wwwdocs repository

[Gerald Pfeifer]

On Wed, 25 Sep 2019, Jonathan Wakely wrote:
> I mean that there's not much value in having my past commits listed as
> coming from various "different" authors:
> 
> Jonathan Wakely <xxx at compsoc.man.ac.uk>
> Jonathan Wakely <xxx at kayari.org>
> Jonathan Wakely <xxx at gmail.com>
> Jonathan Wakely <xxx at gcc.gnu.org>
> Jonathan Wakely <xxx at redhat.com>
> 
> All of those "identities" are the same person, so making all my
> commits use the same Author is arguably correct, even if I was using
> different addresses at different times.

In my case that address really is gerald@pfeifer.com (which I should
keep for the rest of my life).

I noticed today that old commits show up as gerald@gcc.gnu.org, whereas
those from after the conversion show up as gerald@pfeifer.com.  Is there
a way to unify that to gerald@pfeifer.com throughout?

Gerald

Re: git conversion of GCC wwwdocs repository

[Andreas Schwab]

On Okt 20 2019, Gerald Pfeifer wrote:

> I noticed today that old commits show up as gerald@gcc.gnu.org, whereas
> those from after the conversion show up as gerald@pfeifer.com.  Is there
> a way to unify that to gerald@pfeifer.com throughout?

See "MAPPING AUTHORS" in git-shortlog(1).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."