Re: source mgt....[_HAS_ gcc relevance]

Re: source mgt....[_HAS_ gcc relevance]

[Richard Kenner]

    but wouldn't it be nice if that were automated: so a developer could
    hit the "try to test and merge" button before going home for the
    night, coming back in the morning to either a commit email or a list
    of test failures

I'm not sure I like that kind of automation because of the potentially
unknown delay in the testing process (what if the queue that runs the
tests got stuck).  I'd want to be able to know and control exactly
*when* the change went in.

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

	but wouldn't it be nice if that were automated: so a developer
	could hit the "try to test and merge" button before going home
	for the night, coming back in the morning to either a commit
	email or a list of test failures

    I'm not sure I like that kind of automation because of the
    potentially unknown delay in the testing process (what if the
    queue that runs the tests got stuck).  I'd want to be able to know
    and control exactly *when* the change went in.


I think your worry is premature, but understandable.  We're bumping up
against the limits of email.

It's hard to explain abstractions through specific instantiations, but
harder to explain (and be heard explaining) abstractions through
abstract language.  So in forums like this, I think I tend to state
instances, and hope readers form the underlying abstractions for
themselves.  I'm aiming for the SYNC! or "Aha!" experience.

So, I say "going home for the night, coming back in the morning" --
but there's a less specific abstraction behind that I'm pointing at.

Computers keep getting cheaper and faster.  I think in a few short
years, we developers will each have tons of them for day-to-day work.
What Savannah does for the maddening crowd asynchronously and slowly,
you and I will have synchronously and quickly.  (Which is a related
observation, if you think about it.)


-t

Re: source mgt....[_HAS_ gcc relevance]

[Robert Dewar]

> A lot of the thinking behind arch is to scale up and simplify adopting
> practices such as you describe so that they are applied by default to
> pretty much all of the free software (and "open source") projects in
> the world.  With your 35, you have social pressures and the power of
> the employer to enforce restrictions like "run the tests before
> committing to mainline" -- but wouldn't it be nice if that were
> automated: so a developer could hit the "try to test and merge" button
> before going home for the night, coming back in the morning to either
> a commit email or a list of test failures -- and if you _didn't_ have
> to write all your own tools for that automation because they were just
> there already, such that setting up a new project with these
> properties was as easy as creating a project on Savannah currently is
> (or, easier :).

Yes, indeed, automating requirements like this is always desirable.
Although you probably want ways to override requirements in emergencies.

What we find is that the key point is that it must be *easy* to follow
procedures, if it is, then they get followed, if not, no amount of
social pressure can guarantee conformance :-)

Re: source mgt....[_HAS_ gcc relevance]

[Robert Dewar]

> Well, my ideal is that changes to the mainline should occur only
> _after_ they have verifiably passed all the available tests on a wide
> range of platforms (a process that can be fully automated) and the
> changes have passed senior engineer reviews (a process that can be
> facilitated by substantial automated assistance).  Mainlines should
> increase in quality in a strictly monotonic fashion -- that's the
> essence of what "gatekeeper management" is all about.  Neither GCC nor
> lk have that property -- though better tools can do much to put us
> there.  With good tools, the release manager can ultimately be
> replaced by shell scripts.

with GNAT, we let everyone within ACT, which is quite a diverse set of folks
about 35 in all, change anything in the mainline, but we guarantee the
monotonic properly (I agree this is crucial) by enforcing fairly strenuous
requirements on anyone doing a change. No change of any kind (not even
something that is "obviously" safe) is allowed without doing a complete
bootstrap, and running the entire regression suite (which is pretty
comprehensive at this stage) first. Now we only require this on one target
for changes that are expected to be target independent, so it is possible
to have unanticipated hits on other targets. We deal with this by building
the system on all targets every night and running the regression suites on
all targets every night. If the reports in the morning indicate a problem,
then it is all hands on deck to fix the problem.

When we get GNAT properly integrated into GCC, which involves several
things still to be done:

1. We need to get to a release point internally where the GCC 3 based GNAT
passes all regression tests etc. We are close to this, and expecting to
do a beta release in January on selected targets (should include Solaris,
Windows, GNU/Linux, HPUX).

2. We need to get the sources and our internal source procedures more
amenable to GCC style (e.g. we have removed the version numbers from
our sources, and adjusted all our scripts for this change recently).

3. We need to establish the ACATS test suite so that anyone can run it. This
is not as comprehensive as our internal test suite (which is not distributable
since it is mostly proprietary code).

4. We need to set up procedures so we can run and test changes that others
make against our internal test suite.

... then hopefullly we can duplicate at least some of these procedures
in a manner that others outside ACT can follow a similar path. We regard
this kind of automatic testing as absolutely crucial.

> With good tools, the release manager can ultimately be
> replaced by shell scripts.

I don't believe that, based on our experience where we have elaborate
scripts that try to automate everything, but you still need a release
manager to coordinate activities and check that everything is working
as expected.

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

       with GNAT, we let everyone within ACT, which is quite a diverse
       set of folks about 35 in all, change anything in the mainline,
       but we guarantee the monotonic properly (I agree this is
       crucial) by enforcing fairly strenuous requirements on anyone
       doing a change. [....]

Thanks for the report. 

A lot of the thinking behind arch is to scale up and simplify adopting
practices such as you describe so that they are applied by default to
pretty much all of the free software (and "open source") projects in
the world.  With your 35, you have social pressures and the power of
the employer to enforce restrictions like "run the tests before
committing to mainline" -- but wouldn't it be nice if that were
automated: so a developer could hit the "try to test and merge" button
before going home for the night, coming back in the morning to either
a commit email or a list of test failures -- and if you _didn't_ have
to write all your own tools for that automation because they were just
there already, such that setting up a new project with these
properties was as easy as creating a project on Savannah currently is
(or, easier :).


    > With good tools, the release manager can ultimately be
    > replaced by shell scripts.

    I don't believe that, based on our experience where we have
    elaborate scripts that try to automate everything, but you still
    need a release manager to coordinate activities and check that
    everything is working as expected.

Yeah, I said it badly.  Observing gcc, Mark makes lots of judgment
calls and sets focus and picks dates and things like that.  I only
meant that a lot of the mechanical source and repository manipulation
chores can be far better automated than foisting them off onto poor
'ol Zack :)

I have a question about ACT:  How much value do you place in "clean
changesets" and how much would you if they were easier to manipulate?

Feature branches of GCC get smooshed into big merges on mainline, and
I'm wondering if my feeling that keeping changes independent is
shared by others.  Maybe this applies more to the kernel, where
long-lived distribution forks regularly differ in their choices of
changes applied.

It's an interesting question for a rev ctl implementor because, if
maintaining clean changesets is a big deal, then I think you want some
high-level features to help maintain them over time.  For example, if
one change appears on mainline, then months later, a fix related to
that change appears -- then you want to help cherry-pickers associate
those two and combine them.  Another example: under some
circumstances, you want to recognize textually distinct merges that do
nothing more than combine equal sets of changes as, in some sense,
equivalent (as in, I did that same merge as him but in my own way --
even thought I don't have _his_ patch for that merge, don't treat my
tree as if it were missing that patch.)

Second paragraph says:

       A lot of the thinking behind arch is to scale up practices such
       as you describe so that they are applied by default to pretty
       much all of the free software (and "open source") projects in
       the world.

That's part of why I'm pretty frustrated.  I think that is "obviously"
doable and "obviously" generates an (admittedly hard to measure in
conventional ways) ROI on R&D investment in arch.  Nobody seems to
know how to do that in industry, though -- or even how to consider the
correctness of my assessment.  People know how to do things like "buy
Rational".

-t

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

       me:

       > Yeah, I said it badly.  Observing gcc, Mark makes lots of
       > judgment calls and sets focus and picks dates and things like
       > that.

And even there, maybe said it badly again.  I'm really not trying to
sell Mark short.  "Generally keeps on top of things in gcc
developement (no small task)" might be a better description -- i.e.,
not just throwing darts at a calendar and so forth.

-t

Re: source mgt....[_HAS_ gcc relevance]

[Florian Weimer]

Tom Lord <lord@emf.net> writes:

> A lot of the thinking behind arch is to scale up and simplify adopting
> practices such as you describe so that they are applied by default to
> pretty much all of the free software (and "open source") projects in
> the world.  With your 35, you have social pressures and the power of
> the employer to enforce restrictions like "run the tests before
> committing to mainline" -- but wouldn't it be nice if that were
> automated: so a developer could hit the "try to test and merge" button
> before going home for the night, coming back in the morning to either
> a commit email or a list of test failures -- and if you _didn't_ have
> to write all your own tools for that automation because they were just
> there already, such that setting up a new project with these
> properties was as easy as creating a project on Savannah currently is
> (or, easier :).

Well, you can do this using Aegis for a couple of years now, but I
don't see that Aegis is adopted by the free software crowd.  Most of
its members seem to have strong reservations regarding processes which
are enforced by software. ;-)

Re: source mgt....[_HAS_ gcc relevance]

[Momchil Velikov]

>>>>> "Florian" == Florian Weimer <fw@deneb.enyo.de> writes:

    Florian> Well, you can do this using Aegis for a couple of years now, but I
    Florian> don't see that Aegis is adopted by the free software crowd.  Most of
    Florian> its members seem to have strong reservations regarding processes which
    Florian> are enforced by software. ;-)

And rightfully so.  Aegis' model of "change" assignments to developers
doesn't quite fit the volunteer driven projects.

~velco

Re: source mgt....[_HAS_ gcc relevance]

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

Am Die, 2002-12-17 um 09.17 schrieb Momchil Velikov:

> And rightfully so.  Aegis' model of "change" assignments to developers
> doesn't quite fit the volunteer driven projects.

And it's pain in the rear to use. Frankly _I_ want a system consisting 
of _one_ swiss army knife tool and not dotzend of more or less obviously
christened tools cluttering up /usr/bin. The system has to be easy to
explore, not necessarily to setup, and the basic features have to be
easily accessible; cvs and subversion provide that, bk not to that
extend but Aegis is far down the drain in this regard.
I claim that having a CM tools which is trivial to get started with is
important to not scare possible volunteers away, as a rule of a thumb
no more than 3 (better 2) easily explainable steps should be necessary
to retrieve a specific version.

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

       Florian> Well, you can do this using Aegis for a couple of
       Florian> years now, but I don't see that Aegis is adopted by
       Florian> the free software crowd.  Most of its members seem to
       Florian> have strong reservations regarding processes which are
       Florian> enforced by software. ;-)

   velco:

   > And rightfully so.  Aegis' model of "change" assignments to
   > developers doesn't quite fit the volunteer driven projects.

Eh.

The volunteers to some really key projects tend to be either
corporations (or corporate sponsored hackers) or just really
serious-minded individual hackers.

I think audiences like that can appreciate a little bondage and
discipline from their tools if the overall effect is to make their
life easier and more pleasant and their work-product quality higher.
Well, actually, I don't just _think_ this, I _know_ it because that's
part of how GCC works already.

Moreover, those key projects look to me like trend setters.  They can 
raise the bar for other projects by leading by example.

I agree that Aegis isn't quite the right fit for the development
practices we see in free software projects -- but (a) process
enforcement isn't in and of itself the problem (relentless, fixed-form
process enforcement and inflexability is) and (b) I've at least
casually tried to interest Peter Miller into contributing some lessons
from aegis to arch superstructure.  I think it's a good fit.

All: Not to gush, but: The engineering process quality on gcc is so
damn high, especially comparatively.  I feel _very_ self conscious
making so much noise on this list -- so please realize that it's a
very considered decision to do so.  In an ideal universe I'd just be
able to say "Here's arch 1.0, I'm sure you'll like it", but fiscal
reality means I have to advocate earlier in the development cycle.  I
know there's no global GCC budget or anything like that, but there is
a human network here and it does connect to people that can spend
money, with relatively few degrees of separation.

-t

Re: source mgt. requirements solicitation

[Linus Torvalds]

In article <20021215014255.GA1146@doctormoo> you write:
>
>In GCC, we've been known to lose development history when we merge a 
>branch, and merging branches has been incredibly painful.  So I'm not
>sure merging forks is actually harder; merging branches may be. ;-)  

Heh. That's a sad statement about CVS branches in itself.

>Fork merges get submitted as a series of patches (which then need to get 
>approved), and associated ChangeLog entries.  They go in pretty cleanly. 

This is actually not that different from the "old" Linux way, ie the SCM
does _nothing_ for merging stuff. It certainly worked fine for me, and
it's how about half of the Linux developers still work.

The advantage of the SCM-assisted merges is really that when you trust
the other side, it becomes a non-issue.  So to some degree you might as
well think of a SCM-assisted merge as having "write access" to the tree,
except it's a one-time event rather than a continuing process (but
unlike CVS write access it doesn't _need_ to be constant, since both
sides have access to their own SCM structures on their own, and don't
need to merge all the time).

>The fork developer can track his/her own internal change history however 
>he or she likes, but generally will submit an 'expurgated' history for 
>merging, devoid of the false starts, which makes the patches a lot easier 
>to review.  This is in fact an argument in favor of losing 
>development history.  ;-D

We do that with BK too, occasionally. It's sometimes just cleaner to
create a new clone with a "cleaned up" revision history. It's not needed
all that often, but I certainly agree that sometimes you just don't want
to see all the mistakes people initially made.

It's also needed in BK for things like merging from two totally
different repositories - you can't auto-merge just one fix from a
Linux-2.4.x BK tree into a 2.5.x BK tree, for example (when you merge in
BK, you merge _everything_ in the two repositories).  So those have to
be done as patches, kind of like the clean-up thing. 

>>Yet it is the _cheap_ branches that should be the true first-class
>>citizen. Potentially throw-away code that may end up being really 
>>really useful, but might just be a crazy pipe-dream. The experimental 
>>stuff that would _really_ want to have nice source control.
>
>Interestingly, I tend to find that this sort of stuff is exactly what
>*doesn't* need source control; source control simply obscures the 
>process by exposing too much development history, much of which has no 
>relevance to the current version.  Or did you mean code that already 
>works, and is being refined, rather than code in the 'rewrite from 
>scratch every two weeks' stage?

I personally find that _most_ changes by far tend to be fairly small and
simple, and take a few hours or days to do. Yet at the same time, you
want to have access to a lot of the SCM functionality (commit one set of
changes as "phase 1 - preparation", "phase 2 - update filesystems" etc).

At the same time, the tree often doesn't work until all phases are done,
so you do NOT want to commit "phase 1" to the CVS head - and creating a
CVS branch for something that is really not a big project is clearly not
something most people want to do. The pain of the branch is bigger than 
it's worth.

And THIS is where the distributed repository nature of BK really shines.
It's a totally everyday thing, not something odd or special. You can
work in your own repository, with all the SCM tools, and document your
changes as you make them (and undo something if you notice it was
wrong). Yet you do _not_ need to pollute anything that somebody else
might be working on.

And then, when you're ready, you just push your changes to some other
tree (in BK it's an atomic operation to push _multiple_ changesets), and
tell others that you're done.

See? I'm not talking about a big six-month project.  I'm talking about
something that potentially is just a few hours.  You might do your first
cut, and check it into your tree, verify that it works, and then you
might want to go back and make another few check-ins to handle other
cases. 

In gcc terms, let's say that you change the layout of something
fundamental, and you first make sure that the C front-end works. You
check that in and test it (on the C side only) as one stage. Only when
you're happy with that do you even _bother_ to start editing the C++ and
other front-ends.

With distributed trees, it's easy to make these kinds of multi-stage
things. Because nobody else sees what you're doing until you actually
decide to export the thing. With CVS, it's a total _disaster_ to do
this (and the way everybody works is to do all the work _without_ SCM
support, and then try to do one large check-in).

			Linus

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

       The advantage of the SCM-assisted merges is really that when
       you trust the other side, it becomes a non-issue.

It helps even when you don't implicitly trust the other side.

A remote, less-than-implicitly-trusted developer submits a patch.  You
kick it back with comments.   Meanwhile, your mainline has gone on.

Before resubmitting, that developer has to update his patch to reflect
the new head-of-mainline.

If that remote developer has his own repository, but a true,
first-class branch of your mainline, then he can use SCM-assisted
merges to keep his patch up-to-date.

A similar case occurs if you accept his patch, but then there's still
more to be done with it -- further development.  In that case, there's
effectively back-and-forth merging between your mainline and his
remote branch.  "star topology merging" handles exactly that case.

In relation to these features, it's interesting to read the recent
narrowly-on-topic traffic on the gcc list (Mark and Zack's
coordination with everyone else, comments about forming intermediate
merges, and calls for help with testing branches).  I think that some
of the issues around synchronizing work would be somewhat relaxed by
applying these features; intermediate merges would be well
facilitated and _partially_ automated (regardless of access writes of the
authors); testing branches could be made more effectively
automated (again, orthogonally to access rights).

-t

Re: source mgt....[_HAS_ gcc relevance]

[Linus Torvalds]

On Sat, 14 Dec 2002, Tom Lord wrote:
>
>        The advantage of the SCM-assisted merges is really that when
>        you trust the other side, it becomes a non-issue.
>
> It helps even when you don't implicitly trust the other side.

Sorry, you're right, and that's really what I mean.

To give write access to the tree you _really_ have to trust the other
guy. You have to trust him not only today, but you have to trust his
judgement in the long term. Which is why open source projects don't tend
to give write access to CVS trees very lightly, and as a result you tend
to have a fairly small "clique" of trusted developers.

In contrast, for a one-time merge you only have to trust "that merge".
Which is a different thing altogether, and doesn't require any kind of
"implicit trust".

> A remote, less-than-implicitly-trusted developer submits a patch.  You
> kick it back with comments.   Meanwhile, your mainline has gone on.
>
> Before resubmitting, that developer has to update his patch to reflect
> the new head-of-mainline.
>
> If that remote developer has his own repository, but a true,
> first-class branch of your mainline, then he can use SCM-assisted
> merges to keep his patch up-to-date.

Absolutely. And he can still use SCM-assisted merges with his cohorts to
fix whatever issues you brought up:  that's also an important part of
a distributed SCM.

Many people want to check stuff into the CVS tree _not_ because they
really want everybody to see the thing, but because they use the CVS tree
as a way to communicate with a few other people working on the same thing.
That's where the "single main repository" _really_ falls down. CVS tends
to use branches for this, but I do not know a single CVS user who isn't a
bit nervous about creating a new branch. That's true even when you don't
have the "write access" issue.

In contrast, with a good distributed system, if you want to just
communicate with some other people working on the same thing, you can
merge only the _affected_ trees, not the "main" tree. Merge into the main
tree only when you're done and the people involved are happy.

				Linus

Re: source mgt....[_HAS_ gcc relevance]

[Zack Weinberg]

Tom Lord <lord@emf.net> writes:

> In relation to these features, it's interesting to read the recent
> narrowly-on-topic traffic on the gcc list (Mark and Zack's
> coordination with everyone else, comments about forming intermediate
> merges, and calls for help with testing branches).  I think that some
> of the issues around synchronizing work would be somewhat relaxed by
> applying these features; intermediate merges would be well
> facilitated and _partially_ automated (regardless of access writes of the
> authors); testing branches could be made more effectively
> automated (again, orthogonally to access rights).

Oh, no kidding.  I totally agree that a distributed repository system
is the way to go long term, and you'll notice that easy branches was
close to the top of my requirements list.

My comment about the Linux kernel development process was intended as
a comment about the way that project operates, which I happen to have
concerns about, but I'm pretty sure that's the way it has always
operated, independent of version-control system in use.

Linus Torvalds <torvalds@transmeta.com> writes:
> Many people want to check stuff into the CVS tree _not_ because they
> really want everybody to see the thing, but because they use the CVS tree
> as a way to communicate with a few other people working on the same thing.
> That's where the "single main repository" _really_ falls down.

And this is the assumption underlying the part of the kernel process
that I have concerns about.  It's true that mostly only a small number
of people work on any given chunk of a large piece of software, but I
strongly disagree that they should go off by themselves and merge into
the main tree only when they're done, as the default mode of
development.  The effect from everyone else's point of view is that
they pop up and inflict a huge indigestible glob of code on everyone,
which invariably has not been tested thoroughly enough and breaks
stuff.  Just watching linux-kernel suggests that this happens all the
time.  The USAGI patches, for instance, or Rusty's module rewrite.
We've had our share of this in GCC, too; the 'subreg_byte' changes
were maintained at enormous effort separate from the main tree for
_years_ before they accumulated enough evidence that it wouldn't break
anything.

Contrariwise, forcing people to do their work in the main tree means
that they have to do it incrementally, more eyes at least skim the
code, and it naturally gets adequate testing as other people try to do
their own work.  Also, I know that every time I've had to stop and
think how to do a transition incrementally, the end result has been
better off for it.  Al Viro's rewrite of the VFS layer is the obvious
example of this, or Jan Hubicka's jump optimizer overhaul.

As a user of the kernel, I am also concerned about the apparent trend
toward having many variant trees not for development, but to present
divergent sets of features and bugfixes.  It strikes me as a recipe
for user confusion and vendor malfeasance.

zw

Re: source mgt....[_HAS_ gcc relevance]

[Tom Lord]

       It's true that mostly only a small number of people work on any
       given chunk of a large piece of software, but I strongly
       disagree that they should go off by themselves and merge into
       the main tree only when they're done, as the default mode of
       development.


Well, my ideal is that changes to the mainline should occur only
_after_ they have verifiably passed all the available tests on a wide
range of platforms (a process that can be fully automated) and the
changes have passed senior engineer reviews (a process that can be
facilitated by substantial automated assistance).  Mainlines should
increase in quality in a strictly monotonic fashion -- that's the
essence of what "gatekeeper management" is all about.  Neither GCC nor
lk have that property -- though better tools can do much to put us
there.  With good tools, the release manager can ultimately be
replaced by shell scripts.


       As a user of the kernel, I am also concerned about the apparent
       trend toward having many variant trees not for development, but
       to present divergent sets of features and bugfixes.  It strikes
       me as a recipe for user confusion and vendor malfeasance.

Don't pull back -- go Furthur.  There's a lot of good side effects
that can be obtained by giving each _end-customer_ (or nearly so)
their own source fork -- not _only_ feature divergence, but also (and
especially) risk management and emergency preparedness.  Divergent
fixes and bugfixes aren't the end of the world.  On the contrary,
they're a good thing.

Centralizing (in a small number of vendors) binary preparation for
millions (or for thousands of enterprises) is, in and of itself,
"vendor malfeasance".  It creates vulnerabilities, slows down
responsiveness to customer needs, and enables monopolistic "lock-in".

The idea that either of these projects is way ahead of the other in
terms of process is, I think, not quite right.  They're both about
equally far from the ideal.

But sure, I'm describing an ideal point that's on the horizon -- not
a change you should try to make next week.  Just something to stroll 
towards.

-t