Re: git mirror at infradead?

Re: git mirror at infradead?

[Daniel Berlin]

On Tue, Jun 9, 2009 at 3:16 PM, Bernie Innocenti<bernie@codewiz.org> wrote:
> On 06/09/09 16:17, Jason Merrill wrote:
>> Bernie Innocenti wrote:
>>> On 06/07/09 12:40, Ralf Wildenhues wrote:
>>>> Is this mirror an independent conversion from the infradead one (i.e., I
>>>> have to throw away the repo and re-download a full repo?  Or can I reuse
>>>> objects)?
>>>
>>> It's an independent mirror, and I wouldn't recommend switching to it yet.
>>>
>>> There are permissions problems, and I might end up rsyncing the whole
>>> infradead repository rather than fixing things locally.
>>
>> Please please do *not* rsync the infradead repository.  The repository
>> on gcc.gnu.org is set up so that I can switch back and forth between
>> pulling from git and using git-svn directly; the infradead repository is
>> not.
>>
>> For one thing, the infradead repository uses svn://gcc.gnu.org/svn/gcc,
>> which makes it impossible to use git-svn to check in changes; the
>> gcc.gnu.org git repository uses svn+ssh://gcc.gnu.org/svn/gcc, as is
>> right and proper.  Also the remotes are in a different place from where
>> git-svn puts them, though I suppose that's easy enough to adjust when
>> fetching.
>
> I won't re-create the repository from scratch, then.
>
> Though I would still need an updated version of git to enable lots of
> branches and tags without wasting too much hard disk space.
>
> Can a sourceware admin please install (or build) git 1.6.3.x?  If there
> are concerns of breaking other things, I could install a local copy in
> ~/bin.
+overseers

I don't see a problem doing this (we definitely don't want two
versions installed), but there are real live git projects on
sourceware so we should be a bit careful.

Re: git mirror at infradead?

[Ian Lance Taylor]

Daniel Berlin <dberlin@dberlin.org> writes:

> I don't see a problem doing this (we definitely don't want two
> versions installed), but there are real live git projects on
> sourceware so we should be a bit careful.

fche has already installed git 1.6.3.2 in /usr/local/bin on sourceware.
That is now the one you will get if you connect to port "git".  Hope
nothing breaks.

Ian

Re: git mirror at infradead?

[Angela Marie Thomas]

iant@google.com wrote:

> Daniel Berlin <dberlin@dberlin.org> writes:
> 
> > I don't see a problem doing this (we definitely don't want two
> > versions installed), but there are real live git projects on
> > sourceware so we should be a bit careful.
> 
> fche has already installed git 1.6.3.2 in /usr/local/bin on sourceware.
> That is now the one you will get if you connect to port "git".  Hope
> nothing breaks.

Someone should rpm -e the git RPMs as well.

--Angela

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

On 06/10/09 02:43, Ian Lance Taylor wrote:
> fche has already installed git 1.6.3.2 in /usr/local/bin on sourceware.
> That is now the one you will get if you connect to port "git".  Hope
> nothing breaks.

Thanks.

I made a few changes that hopefully won't compromise existing clones:

0) Since when Daniel disabled his cron job to update the repository,
   mine had not actually been running because the crontab line was
   commented out.  I enabled it.

1) Set UseDeltaBaseOffset=true for slightly smaller packs
   The downside is that we loose compatibility with git versions
   older than 1.4.4.  Moreover, people fetching from dumb protocols
   will probably have to refetch from scratch.

2) Remove the local checkout and configure the repository as
   "bare=true"

3) I stupidly ran a "git gc" on the repository without specifying
   any parameters, which made the pack jump to a whopping 3.4GB.
   People fetching over git-aware protocols shouldn't notice
   anything unusual except, maybe, longer server-side packing time.
   Those stuck with http:// will have a bad surprise. /me ducks.
   I've now configured the default window and depth both to 100,
   and ran another repack from scratch which will take a long,
   long, long, long time.

4) I noticed we weren't actually doing "git update-server-info"
   anywhere, which means that those stuck with http:// are
   probably unable to be annoyed by my careless mistake anyway.
   I've now added it as the last step of my cron job
   ~bernie/bin/git-update-toolchain.sh

5) The repository was configured with HEAD pointing at the local branch
   "master", which git-svn doesn't know (or care) about.  To make
   changes appear there too, Daniel's update script did a "git rebase"
   after every fetch, which does not work in bare repositories.
   Now I made HEAD point at branch "trunk" instead, and make "trunk"
   just be an indirect ref for "remotes/trunk", which is what git-svn
   actually updates. Hopefully this change won't break existing
   checkouts, but one might still have to perform some manual trickery
   to rename their local branch "master" to "trunk", or map it to
   remotes/trunk in their fetch configuration.

6) As Daniel said, we are indeed already mirroring all branches from
   the SVN repository.  But those go to remotes/foo rather than foo,
   which most likely *your* git fetches aren't configured to consider.
   And gitweb doesn't show them either.  As a workaround, I manually
   added a few local branches pointing directly to the remote ones.
   I arbitrarily chose to export all those gcc-4_x-branch tags, but
   we could add them all with a script, and maybe rename them to
   cull that "-branch" postfix which has been quite redundant since
   when we switched to svn.

I sincerely apologize if these changes happen to hamper someone's
productivity for a while.  Cheer up!  Your temporary suffering is
for the long-term good! :-)

Local to remote branch mapping is one of the least understood design
aspects in git [*], and probably the steepest point of git's steep
learning curve.  It is especially unfortunate that git-svn exposes
it even more than usual.

[*] the git haters would simply call it "lack of design in git"

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Daniel Berlin]

On Thu, Jun 11, 2009 at 3:07 AM, Bernie Innocenti<bernie@codewiz.org> wrote:
> On 06/10/09 02:43, Ian Lance Taylor wrote:
>> fche has already installed git 1.6.3.2 in /usr/local/bin on sourceware.
>> That is now the one you will get if you connect to port "git".  Hope
>> nothing breaks.
>
> Thanks.
>
> I made a few changes that hopefully won't compromise existing clones:
>
> 0) Since when Daniel disabled his cron job to update the repository,
>   mine had not actually been running because the crontab line was
>   commented out.  I enabled it.
>
> 1) Set UseDeltaBaseOffset=true for slightly smaller packs
>   The downside is that we loose compatibility with git versions
>   older than 1.4.4.  Moreover, people fetching from dumb protocols
>   will probably have to refetch from scratch.
>
> 2) Remove the local checkout and configure the repository as
>   "bare=true"
>

Yeah, this i had forgotten to do ;)

> 3) I stupidly ran a "git gc" on the repository without specifying
>   any parameters, which made the pack jump to a whopping 3.4GB.
>   People fetching over git-aware protocols shouldn't notice
>   anything unusual except, maybe, longer server-side packing time.
>   Those stuck with http:// will have a bad surprise. /me ducks.
>   I've now configured the default window and depth both to 100,
>   and ran another repack from scratch which will take a long,
>   long, long, long time.

It may be faster for my to rsync it to a 32 core machine, pack it,
then rsync it back now that delta compression is threaded.
Does it get large enough speedups these days to be worth it?

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

On 06/11/09 14:03, Daniel Berlin wrote:
> It may be faster for my to rsync it to a 32 core machine, pack it,
> then rsync it back now that delta compression is threaded.
> Does it get large enough speedups these days to be worth it?

It's done, and the pack came out 553MB.

Perhaps packing with an even larger window and depth of 250 might
squeeze a few more MBs off...  Worth a try :-)

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

On 06/11/09 15:18, Bernie Innocenti wrote:
> On 06/11/09 14:03, Daniel Berlin wrote:
>> It may be faster for my to rsync it to a 32 core machine, pack it,
>> then rsync it back now that delta compression is threaded.
>> Does it get large enough speedups these days to be worth it?
> 
> It's done, and the pack came out 553MB.
> 
> Perhaps packing with an even larger window and depth of 250 might
> squeeze a few more MBs off...  Worth a try :-)

BTW, I'd be curious to see how much 32 cores can speed up packing.
By default, git-repack autodetects the number of CPUs and spawns threads
accordingly, so it shouldn't require any tuning.  You just need a recent
version of git implementing this optimization.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Jason Merrill]

On 06/11/2009 03:07 AM, Bernie Innocenti wrote:
> 6) As Daniel said, we are indeed already mirroring all branches from
>     the SVN repository.  But those go to remotes/foo rather than foo,
>     which most likely *your* git fetches aren't configured to consider.

Mine are.  I ignore all heads in gcc.git, and just map its remotes into 
my remotes by manually specifying remote.origin.fetch.  Not very pretty, 
but it seems to produce the optimal result.  See my stuff in the lower 
section of http://gcc.gnu.org/wiki/GitMirror for more details.

Jason

Re: git mirror at gcc.gnu.org

[Rafael Espindola]

> Mine are.  I ignore all heads in gcc.git, and just map its remotes into my
> remotes by manually specifying remote.origin.fetch.  Not very pretty, but it
> seems to produce the optimal result.  See my stuff in the lower section of
> http://gcc.gnu.org/wiki/GitMirror for more details.

It fails with

$ git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/origin/*'
$ git fetch
fatal: refs/remotes/origin/gcc-4_0-branch tracks both
refs/remotes/gcc-4_0-branch and refs/heads/gcc-4_0-branch

> Jason
>

Cheers,
-- 
Rafael Avila de Espindola

Google | Gordon House | Barrow Street | Dublin 4 | Ireland
Registered in Dublin, Ireland | Registration Number: 368047

Re: git mirror at gcc.gnu.org

[Jason Merrill]

On 06/15/2009 10:28 AM, Rafael Espindola wrote:
> It fails with
>
> $ git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/origin/*'

That's not my section; my contribution starts at

Alternative git-svn procedure (Jason Merrill)

Jason

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

On 06/15/09 16:28, Rafael Espindola wrote:
>> Mine are.  I ignore all heads in gcc.git, and just map its remotes into my
>> remotes by manually specifying remote.origin.fetch.  Not very pretty, but it
>> seems to produce the optimal result.  See my stuff in the lower section of
>> http://gcc.gnu.org/wiki/GitMirror for more details.
> 
> It fails with
> 
> $ git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/origin/*'
> $ git fetch
> fatal: refs/remotes/origin/gcc-4_0-branch tracks both
> refs/remotes/gcc-4_0-branch and refs/heads/gcc-4_0-branch

Perhaps I should remove those "friendly" refs pointing at the remote
branches?  Or can we find a better alternative?  Their use was to make a
few frequently used branches readily visible in gitweb and with a simple
clone.

Perhaps git-svn could be configured to map svn branches directly to the
local namespace instead of remotes/ ?

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Jason Merrill]

On 06/15/2009 01:22 PM, Bernie Innocenti wrote:
> On 06/15/09 16:28, Rafael Espindola wrote:
>> It fails with
>>
>> $ git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/origin/*'
>> $ git fetch
>> fatal: refs/remotes/origin/gcc-4_0-branch tracks both
>> refs/remotes/gcc-4_0-branch and refs/heads/gcc-4_0-branch
>
> Perhaps I should remove those "friendly" refs pointing at the remote
> branches?  Or can we find a better alternative?  Their use was to make a
> few frequently used branches readily visible in gitweb and with a simple
> clone.

It makes sense to me to have the friendly refs so that the simple case 
(clone, don't try to use svn directly) works easily.

What I'm doing doesn't have any problem with them.  Rafael was following 
older instructions written by someone else.  When I started editing that 
page, I put my suggestions at the bottom because I was fairly new to 
git.  Now I feel more confident that what I'm doing is right (or at 
least better) so I think I'll remove the old instructions.

But what are oldmaster/pre-globals-git/restrict-git?  And why have both 
"master" and "trunk" as heads?

> Perhaps git-svn could be configured to map svn branches directly to the
> local namespace instead of remotes/ ?

It could, but it seems unnecessary.

Jason

Re: git mirror at gcc.gnu.org

[Daniel Berlin]

On Tue, Jun 16, 2009 at 10:17 AM, Jason Merrill<jason@redhat.com> wrote:
> On 06/15/2009 01:22 PM, Bernie Innocenti wrote:
>>
>> On 06/15/09 16:28, Rafael Espindola wrote:
>>>
>>> It fails with
>>>
>>> $ git config --add remote.origin.fetch
>>> '+refs/remotes/*:refs/remotes/origin/*'
>>> $ git fetch
>>> fatal: refs/remotes/origin/gcc-4_0-branch tracks both
>>> refs/remotes/gcc-4_0-branch and refs/heads/gcc-4_0-branch
>>
>> Perhaps I should remove those "friendly" refs pointing at the remote
>> branches?  Or can we find a better alternative?  Their use was to make a
>> few frequently used branches readily visible in gitweb and with a simple
>> clone.
>
> It makes sense to me to have the friendly refs so that the simple case
> (clone, don't try to use svn directly) works easily.
>
> What I'm doing doesn't have any problem with them.  Rafael was following
> older instructions written by someone else.  When I started editing that
> page, I put my suggestions at the bottom because I was fairly new to git.
>  Now I feel more confident that what I'm doing is right (or at least better)
> so I think I'll remove the old instructions.
>
> But what are oldmaster/pre-globals-git/restrict-git?

pre-globals-git and restrict-git were test branches from an old mostly
broken version. ;)

They are dead and if someone wants to manually remove the refs, go for it.
No idea what oldmaster is.

>  And why have both
> "master" and "trunk" as heads?
>

See "broken" ;)

>> Perhaps git-svn could be configured to map svn branches directly to the
>> local namespace instead of remotes/ ?
>
> It could, but it seems unnecessary.
>
> Jason
>

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

Daniel Berlin wrote:
> pre-globals-git and restrict-git were test branches from an old mostly
> broken version. ;)
> 
> They are dead and if someone wants to manually remove the refs, go for it.
> No idea what oldmaster is.

Done.


>>  And why have both
>> "master" and "trunk" as heads?
> 
> See "broken" ;)

Yes, but which one should die?  Will cloners get confused by a
repository where the master branch is missing?

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Jason Merrill]

I also notice a few remotes with @ in them like milepost-branch@129596 
which seem to be git-svn artifacts.

Jason

Re: git mirror at gcc.gnu.org

[Jason Merrill]

On 06/16/2009 03:51 PM, Bernie Innocenti wrote:
> Yes, but which one should die?  Will cloners get confused by a
> repository where the master branch is missing?

clone will get whatever branch is currently active in the cloned 
repository, doesn't matter what it's called.

Currently master is out of date, so it's worse than nothing.

Any thoughts on what to do about the non-branch directories under 
branches/?  The complete set is ARM apple csl dead gcj ibm ix86 suse 
ubuntu.  The only solution I can think of would be to specifically 
enumerate which branches we want to mirror in git with separate git-svn 
fetch lines rather than use the branches= line.

Jason

Re: git mirror at gcc.gnu.org

[Bernie Innocenti]

On 06/20/09 04:59, Jason Merrill wrote:
> clone will get whatever branch is currently active in the cloned
> repository, doesn't matter what it's called.
> 
> Currently master is out of date, so it's worse than nothing.

Ok, culled.


> Any thoughts on what to do about the non-branch directories under
> branches/?  The complete set is ARM apple csl dead gcj ibm ix86 suse
> ubuntu.  The only solution I can think of would be to specifically
> enumerate which branches we want to mirror in git with separate git-svn
> fetch lines rather than use the branches= line.

Don't we want to mirror all branches?  Enumerating them all would be a
lot of work...

Perhaps we could report this missing feature to the git-svn folks and
ask them to fix it for us?

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: git mirror at gcc.gnu.org

[Jason Merrill]

On 06/20/2009 04:38 AM, Bernie Innocenti wrote:
> On 06/20/09 04:59, Jason Merrill wrote:

>> Any thoughts on what to do about the non-branch directories under
>> branches/?  The complete set is ARM apple csl dead gcj ibm ix86 suse
>> ubuntu.  The only solution I can think of would be to specifically
>> enumerate which branches we want to mirror in git with separate git-svn
>> fetch lines rather than use the branches= line.
>
> Don't we want to mirror all branches?  Enumerating them all would be a
> lot of work...

I was thinking to do that programatically.  i.e.

git config --unset svn-remote.svn.branches
for f in `svn ls svn://gcc.gnu.org/svn/gcc/branches|egrep -v 
'^(ARM|apple|csl|dead|gcj|ibm|ix86|redhat|suse|ubuntu)/'`; do
   f=${f%/}
   git config --get-all svn-remote.svn.fetch |
     fgrep "branches/$f:" > /dev/null ||
     git config --add svn-remote.svn.fetch branches/$f:refs/remotes/$f
done
for d in ARM apple csl dead gcj ibm ix86 redhat suse ubuntu; do
   for f in `svn ls svn://gcc.gnu.org/svn/gcc/branches/$d`; do
     f=$d/${f%/}
     git config --get-all svn-remote.svn.fetch |
       fgrep "branches/$f:" > /dev/null ||
       git config --add svn-remote.svn.fetch branches/$f:refs/remotes/$f
   done
done

> Perhaps we could report this missing feature to the git-svn folks and
> ask them to fix it for us?

I'm not sure it's possible in general to tell whether a particular 
directory in SVN is a branch or not.

Jason

Re: git mirror at gcc.gnu.org

[Jason Merrill]

Incidentally, I notice that branches in subdirectories of branches/ are 
still broken; i.e. ARM apple redhat suse ubuntu, maybe others.  I give 
instructions on the GitMirror page for how to deal with that, but I 
wonder if it's possible to set it up properly on the mirror instead.

Jason