Re: source mgt. requirements solicitation

[Linus Torvalds <torvalds@transmeta.com>]

[ See the blurb about OpenCM at the end. ]

In article <87isy2mj1y.fsf@egil.codesourcery.com>,
Zack Weinberg  <zack@codesourcery.com> wrote:
>
>Linux is a large project - 4.3 million lines of code - but only one
>person has commit privileges on the official tree, for any given
>release branch.

No. That's not how it works.

Linux, unlike _every_ other project I know of, has always actively
encouraged "personal/vendor branches", and that is in fact how 99% of
all development has happened.

Most development happens in trees that have _nothing_ to do with the
official tree.  To me, the whole CVS model (many branches in one
centralized repository) is just incredibly broken, and you should
realize that that isn't how Linux has ever worked. 

My tree is often called the "official" tree, but what it really is is
just a base tree that many people maintain their own forks from. 
This is fundamentally _more_ scalable than the CVS mess that is gcc,
since it much more easily allows for very radical branches that do not
need any centralized permission from me. 

Think of it this way: in gcc, the egcs split was a very painful thing.  
In Linux, those kinds of splits (people doing what they think is right,
_without_ support from the official maintainers) is how _everything_ gets
done.  Linux is a "constantly forking" project, and that's how development
very fundamentally happens.

And a fork is a lot more scalable than a branch.  It's also a lot more
powerful: it gives _full_ rights to the forker.  That implies that a
forked source tree should be a first-class citizen, not just something
that was copied off somebody elses CVS tree.  The BitKeeper "clone"
thing is a beautiful implementation of the Linux development model. 

>	  No matter how good their tools are, this cannot be
>expected to scale, and indeed it does not.

Sorry, but you're wrong.  Probably simply because you're too used to the
broken CVS model.

I would like to point out that Linux development has scaled a lot better
than gcc, to a larger source base (it's 5+ M lines) with much more
fundamental programming issues (concurrency etc).  I will bet you that
the Linux kernel merges are a lot bigger than the gcc ones, that
development happens faster, and that there are more independent
developers working on their own versions of Linux than there are of gcc. 

There aren't just a handful of branches, there are _hundreds_. Many of
them end up not being interesting, or ever necessarily merged back. And
_none_ of them required write access to my tree.

I'd also like to point out that Linux has _never_ had a flap like the
gcc/egcs/emacs/xemacs splits.  Exactly because of the _much_ more
scalable approach of just fundamentally always having had a distributed
development model that allows _anybody_ to contribute easily, instead of
having a model that makes certain people have "special powers". 

In short, _my_ tree is _not_ the same thing as the gcc CVS sources.

>				  I have not actually
>measured it, but the appearance of the traffic on linux-kernel is that
>Linus drops patches on the floor just as often as he did before he
>started using Bitkeeper.

Measure the number of changes accepted, and I bet the Linux kernel
approach had an order of magnitude more changes than gcc has _ever_ had.
Even before using Bitkeeper.

The proof is in the pudding - care to compare real numbers, and compare
sizes of weekly merged patches? I bet gcc will come in _far_ behind.

>		  However, Bitkeeper facilitates other people
>maintaining their own semi-official versions of the tree, in which
>some of these patches get sucked up.  That is bad.

No. Have you ever used Bitkeeper? Really _used_ it?

I've used both bitkeeper and CVS (I refuse to touch CVS with a ten-foot
pole for my "fun" projects, but I've used CVS for big projects at work),
and I can tell you, CVS doesn't even come _close_.  Not even with
various wrapper helper tools to make things like CVS branches look even
remotely palatable. 

The part that you're missing, simply because you've probably used CVS
for too long, is the _distributed_ nature of Bitkeeper, and of Linux
development.  Repeat after me: "There is no single tree".  Everything is
distributed. 

Any source control system that has "write access" issues is
fundamentally broken in my opinion.  Your repository should be _yours_,
and nobody elses.  There is no "write access".  There is only some way
to expedite a merge between two repositories.  The source control
management should make it easy for you to export your changes to other
repositories.  In fact, it should make it easy for you to have many
different repositories - for different things you're working on. 

Bitkeeper does this very well.  It's _the_ reason I use bitkeeper.  BK
does other things too, but they all pale to the _fundamental_ idea of
each repository being a thing unto itself, and having no stupid
"branches", but simply having truly distributed repositories.  Some
people think that is a "offline" feature, but nothing could be further
from the truth.  The _real_ issue about independent repositories is that
it makes it possible to do truly independent development, and makes
notions like branches such an outdated idea. 

Projects like Subversion never seem to have really _understood_ the
notion of true distributed repositories. And by not understanding them,
like you they miss the whole point of truly scalable development.
Development that scales _past_ the notion of one central repository.

>Possibly I am too pessimistic.

No. You're not pessimistic, you just don't _understand_.

You don't have to believe me.  Believe the numbers.  Look at which
project gets more done.  And realize that even before Linux used
Bitkeeper, it used the truly distributed _model_.  The model is
independent from what SCM you use, although some SCMs obviously cannot
support some models (and CVS in particular forces its users to use a
particularly broken model). 

Btw, I realize that there's no way in hell gcc will use bitkeeper.  I'm
not trying to push that.  I'm just hoping that if gcc does change to
something smarter than CVS, it would change to something that is truly
distributed, and doesn't have that broken "branch" notion, or the notion
of needing write permissions to some stupid central repository in order
to enjoy the idea of SCM. 

Looking at the current projects out there, the only one that looks like
it has more than half a clue is "OpenCM".  It doesn't seem to really do
the distributed thing right, but at least from what I've seen it looks
like they have the right notions about doing it.

The OpenCM project seems to still believe that distribution is just
about "disconnected commits" rather than understanding that if you do
distributed repositories right you shouldn't have branches at all
(instead of a branch, you should just have a _different_ repository),
but they at least seem to understand the importance of true
distribution.  I hope gcc developers are giving it a look. 

			Linus