Merging lessons

Merging lessons

[Rick Moseley]

Hi all,

I have been asked by Tom to distill some of my "lessons learned" while 
doing the merges the past couple of months.  Although I have learned a 
lot of lessons here, I'm sure Jan has some more experience and I hope he 
will add to what I have written here.  Maybe we can put together some 
guidelines on a wiki page when we consolidate all thoughts on this 
subject.  I know a lot of these are intuitive, but I feel they should be 
enumerated here.

1)  Make sure you start from the branch you want to merge back into; 
make your branch

      I had a lot of pain trying to merge modules back into branches 
that seemed really out of sync with the
      branch I was merging it into.  This pain increases almost 
exponentially as the size of the module
      increases.

2)  Do a rebase at least once a week

     For the same reasons as no. 1, this especially is true if work is 
being done on commonly modded
     modules such as gdb/breakpoint.c, which is many thousands of lines.

3)  We need some policy on ChangeLogs.

     I haven't seen a ChangeLog merge yet that went smoothly.  Using git 
to "attempt" to merge
     the files produces a file that is most painful to merge.

4)  Do a rebase before editing configure files such as Makefile.*, 
configure.*, etc. as these files are
     difficult to merge.

     The configure files rank right up there, IMHO, with ChangeLog as 
being difficult to merge,
     make sure you have the latest version before making any changes.



These are just a few thoughts I had on the merging process.  Jan, I hope 
you add much more to this
as you have had to master the merging of gdb code for some time now.  
Any pointers/suggestions
would be most appreciated here.

Thanks,


Rick

Re: [Archer] Merging lessons

[Joel Brobecker]

> 3)  We need some policy on ChangeLogs.
> 
>     I haven't seen a ChangeLog merge yet that went smoothly.  Using git 
> to "attempt" to merge
>     the files produces a file that is most painful to merge.

What we do at AdaCore is use our own ChangeLog file (ChangeLog.GNAT
FWIW).  Similarly, when I work on my own personal branch, I sometimes
use my own ChangeLog file called whatever. However, now that I'm using
git more and more often, I've found the use of ChangeLogs to simply
be redundant and limitative - I just put the ChangeLog entry as my
revision history, and git is fast enough to produce a "ChangeLog"
on the fly... This worked really well for me while I was trying
to contribute the changes for 64bit Windows.

-- 
Joel

Re: Merging lessons

[Tom Tromey]

>>>>> "Rick" == Rick Moseley <rmoseley@redhat.com> writes:

Rick> 2)  Do a rebase at least once a week

I'd prefer people merge from master, rather than rebase.  IIUC,
rebasing doesn't work too well in a shared repository, but merging
does.

I will endeavor to merge more frequently.

To anybody who does a future topic->release merge, please feel free to
harangue people into doing their master->topic merges as the right
moment.

Rick> The configure files rank right up there, IMHO, with ChangeLog as
Rick> being difficult to merge, make sure you have the latest version
Rick> before making any changes.

We can treat the generated files as outputs -- you don't have to merge
them at all, just regenerate them by running the tools.

What I do is download the proper auto* versions from ftp.gnu.org, then
install them into a fresh directory.  Then I put the new $prefix/bin
in my $PATH whenever I need to run auto* on gdb.

It is pretty important to get the exact upstream version -- the tools
are version-sensitive and even if your distro does happen to package
that version, there is a fair chance there is an output-affecting
distro-specific patch in there.

Tom

Re: [Archer] Merging lessons

[Tom Tromey]

>>>>> "Joel" == Joel Brobecker <brobecker@adacore.com> writes:

>> 3)  We need some policy on ChangeLogs.
>> 
>> I haven't seen a ChangeLog merge yet that went smoothly.  Using git 
>> to "attempt" to merge
>> the files produces a file that is most painful to merge.

Joel> What we do at AdaCore is use our own ChangeLog file (ChangeLog.GNAT
Joel> FWIW).  Similarly, when I work on my own personal branch, I sometimes
Joel> use my own ChangeLog file called whatever. However, now that I'm using
Joel> git more and more often, I've found the use of ChangeLogs to simply
Joel> be redundant and limitative - I just put the ChangeLog entry as my
Joel> revision history, and git is fast enough to produce a "ChangeLog"
Joel> on the fly... This worked really well for me while I was trying
Joel> to contribute the changes for 64bit Windows.

We should pick one of these and go with it.

At the moment I don't think I have a preference.  My main concerns are
that we have decent change logging, and that we reduce the pain of
merging.  Either of these seems to fit the bill.

Someone want to make a proposal?

Tom

Re: Merging lessons

[Rick Moseley]

Tom Tromey wrote:
>>>>>> "Rick" == Rick Moseley <rmoseley@redhat.com> writes:
>>>>>>             
>
> Rick> 2)  Do a rebase at least once a week
>
> I'd prefer people merge from master, rather than rebase.  IIUC,
> rebasing doesn't work too well in a shared repository, but merging
> does.
>
> I will endeavor to merge more frequently.
>
> To anybody who does a future topic->release merge, please feel free to
> harangue people into doing their master->topic merges as the right
> moment.
>
> Rick> The configure files rank right up there, IMHO, with ChangeLog as
> Rick> being difficult to merge, make sure you have the latest version
> Rick> before making any changes.
>
> We can treat the generated files as outputs -- you don't have to merge
> them at all, just regenerate them by running the tools.
>   
I should have been more specific here.  As Jan pointed out in the 
meeting Wednesday you can regenerate the "configure" files.  What I was 
referring to here were the files that create the "configure" and 
"Makefile" files such as "configure.*", "Makefile.*", and "*.m4" files.  
Some of these files are pretty large and get cumbersome quickly.
> What I do is download the proper auto* versions from ftp.gnu.org, then
> install them into a fresh directory.  Then I put the new $prefix/bin
> in my $PATH whenever I need to run auto* on gdb.
>
> It is pretty important to get the exact upstream version -- the tools
> are version-sensitive and even if your distro does happen to package
> that version, there is a fair chance there is an output-affecting
> distro-specific patch in there.
>
> Tom
>   

Re: Merging lessons

[Jan Kratochvil]

On Fri, 16 Jan 2009 01:33:12 +0100, Tom Tromey wrote:
> It is pretty important to get the exact upstream version -- the tools
> are version-sensitive and even if your distro does happen to package
> that version, there is a fair chance there is an output-affecting
> distro-specific patch in there.

Using these custom rpms for this purpose:
  http://people.redhat.com/jkratoch/autosw/
  http://people.redhat.com/jkratoch/autosw/autoconf259-2.59-12_nointrinsic.noarch.rpm
  http://people.redhat.com/jkratoch/autosw/automake19-1.9.6-2.1.noarch.rpm

Stepan Kasal was discouraging me from the need for distro-included packaged
autoconf-2.59 (as is packaged autoconf-2.13) as for 2.13 the port was too
difficult while for 2.59->current the port should be easy and it is better to
upgrade upstream (FSF GDB=sourceware repo) to the latest autoconf.

Is there a preliminary agreement for a FSF GDB+sourceware tree HEAD port to
autoconf-2.63?


Regards,
Jan

Re: [Archer] Merging lessons

[Jan Kratochvil]

On Fri, 16 Jan 2009 01:38:02 +0100, Tom Tromey wrote:
> At the moment I don't think I have a preference.  My main concerns are
> that we have decent change logging, and that we reduce the pain of
> merging.  Either of these seems to fit the bill.

I find unclear not where but even what should be maintained as Archer
changelogs:

As Archer is based on the continuously changing FSF GDB and the FSF changelogs
are detailed enough to require an update on many merge conflict cases I find
them difficult to maintain and use only the single sentence patch goal.  FSF
changelog I create only at the point of submitting it for FSF GDB.

In fact I find Tom also does the same sometimes, such as for
  3ba4f9e029fb84916f6e16a4d4dfd941796669ce
  fix off-by-one error when concatenating strings

Should Archer submit a full FSF changelog history of the branch being
checked-in FSF GDB or just the final patch changes?  Or should be maintained
the FSF changelog style for Archer and then written a new FSF changelog entry
for the FSF GDB merge?


Regards,
Jan

Re: Merging lessons

[Daniel Jacobowitz]

On Fri, Jan 16, 2009 at 03:33:17PM +0100, Jan Kratochvil wrote:
> Is there a preliminary agreement for a FSF GDB+sourceware tree HEAD port to
> autoconf-2.63?

The problem is not so much GDB+sourceware as GCC; given the shared
top level files, we have too many configure scripts that need to be
updated all at once.  No one objects in principle.  I believe Ralf
W. was working on it for GCC?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Merging lessons

[Jan Kratochvil]

Hi,

On Thu, 15 Jan 2009 20:39:29 +0100, Rick Moseley wrote:
> I'm sure Jan has some more experience and I hope he will add to what I have
> written here.

(a1) Run the testsuite on all the supported arches.  Do not run the testsuite
     on shared boxes as the GDB testsuite can (could) crash both ptrace and
     utrace kernels.  Also run the testsuite on the right kernel version as at
     least threads debugging support differs on older kernels.  Do not use
     KVM.  As a summary do not use Koji / any build farms for the testsuite.

(a2) Even during development I use for ./configure many options from:
     http://cvs.jankratochvil.net/viewcvs/nethome/bin/errs1?rev=HEAD
     IIRC suggested by Tom: --enable-targets=all
       (our change could break a target not being build on our host)
     -lmcheck (memory corruption checking cheap enough for daily use)
     -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector (rpmbuilt-like runtime checks)

(a3) During larger rebase do not try to save time by hoping it will build out
     of the box on any of the arches.  Each arch will require at least one
     minor fixup by hand anyway.

(a4) Try to easily drop the patches which got imported to the base sourcetree
     but be sure to not to drop something not imported there.  Not each patch
     has a testcase as occasionally it is too difficult to make it.  In Fedora
     I put the backported patches to an single file which one can easily drop,
     "gdb-6.X-upstream.patch", but in fact many patches get checked in the
     branched sources before being accepted for the base sourcetree and later
     they get accepted there in a modified/different form.  I am not aware of
     any universal foolproof approach, one needs to know+track the patches.

(a5) Run even the biarch (-m32/-m64) and/or PIE (-fPIE) testsuite/testcases.
     I will not write more about it as it is not 100% working for me so far.


Regards,
Jan

Re: [Archer] Merging lessons

[Tom Tromey]

>>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:

Jan> I find unclear not where but even what should be maintained as Archer
Jan> changelogs:

Jan> As Archer is based on the continuously changing FSF GDB and the
Jan> FSF changelogs are detailed enough to require an update on many
Jan> merge conflict cases I find them difficult to maintain and use
Jan> only the single sentence patch goal.  FSF changelog I create only
Jan> at the point of submitting it for FSF GDB.

Yeah.  Maintaining a ChangeLog entry over time is a real pain.

Jan> In fact I find Tom also does the same sometimes, such as for
Jan>   3ba4f9e029fb84916f6e16a4d4dfd941796669ce
Jan>   fix off-by-one error when concatenating strings

On the charset branch I've been experimenting with a different
approach -- writing a big ChangeLog entry and then updating it as I
make changes.  I can't really recommend it.

Jan> Should Archer submit a full FSF changelog history of the branch
Jan> being checked-in FSF GDB or just the final patch changes?  Or
Jan> should be maintained the FSF changelog style for Archer and then
Jan> written a new FSF changelog entry for the FSF GDB merge?

I think next time I am going to just have a separate ChangeLog.branch
on my branch, and then rewrite it into a single ChangeLog whenever I
push upstream.

Tom

Re: [Archer] Merging lessons

[Thiago Jung Bauermann]

[I was on vacation, so I'm late in this discussion...]

El jue, 15-01-2009 a las 17:38 -0700, Tom Tromey escribió:
> >>>>> "Joel" == Joel Brobecker <brobecker@adacore.com> writes:
> 
> >> 3)  We need some policy on ChangeLogs.
> >> 
> >> I haven't seen a ChangeLog merge yet that went smoothly.  Using git 
> >> to "attempt" to merge
> >> the files produces a file that is most painful to merge.
> 
> Joel> What we do at AdaCore is use our own ChangeLog file (ChangeLog.GNAT
> Joel> FWIW).  Similarly, when I work on my own personal branch, I sometimes
> Joel> use my own ChangeLog file called whatever. However, now that I'm using
> Joel> git more and more often, I've found the use of ChangeLogs to simply
> Joel> be redundant and limitative - I just put the ChangeLog entry as my
> Joel> revision history, and git is fast enough to produce a "ChangeLog"
> Joel> on the fly... This worked really well for me while I was trying
> Joel> to contribute the changes for 64bit Windows.
> 
> We should pick one of these and go with it.
> 
> At the moment I don't think I have a preference.  My main concerns are
> that we have decent change logging, and that we reduce the pain of
> merging.  Either of these seems to fit the bill.
> 
> Someone want to make a proposal?

+1000 for Joel's idea of putting the ChangeLog entry in the commit
message.

I rebase, cherry-pick, reorder and rewrite my commits very often, to
create patches for upstream submission and to organize my work before
pushing to the python branch. Every single time I do that, I have to
solve a conflict in a ChangeLog file. Not having to clean such mess
would definitely make my life simpler.

It's easy enough to collect the relevant changelog entries from the
commit messages when producing a patch for upstream.
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center

Re: [Archer] Merging lessons

[Jim Meyering]

Tom Tromey <tromey@redhat.com> wrote:
...
> Jan> Should Archer submit a full FSF changelog history of the branch
> Jan> being checked-in FSF GDB or just the final patch changes?  Or
> Jan> should be maintained the FSF changelog style for Archer and then
> Jan> written a new FSF changelog entry for the FSF GDB merge?
>
> I think next time I am going to just have a separate ChangeLog.branch
> on my branch, and then rewrite it into a single ChangeLog whenever I
> push upstream.

Hi Tom,

I find maintaining a ChangeLog file to be too burdensome.  Using a VC'd
ChangeLog makes it so what should be trivial day-to-day operations
(e.g., reordering change-sets, rebasing) are almost guaranteed to
provoke merge conflicts.  In maintaining coreutils, I compromised: the
ChangeLog still exists, but only virtually, as a sequence of log entries.
However, there's a pretty good script to convert commit-log entries into
ChangeLog entries.  And I write commit log entries knowing how they'll
be converted.  Simple rules and incomplete commit hooks help ensure that
the result is reasonable.

At 'make dist' time, a ChangeLog file is automatically generated from
commit logs and inserted into the resulting tarball.  To do that, I use
this rule in Makefile.am:

gen_start_date = 2008-02-08
.PHONY: gen-ChangeLog
gen-ChangeLog:
        if test -d .git; then                                           \
          $(top_srcdir)/build-aux/gitlog-to-changelog                   \
            --since=$(gen_start_date) > $(distdir)/cl-t;                \
          rm -f $(distdir)/ChangeLog;                                   \
          mv $(distdir)/cl-t $(distdir)/ChangeLog;                      \
        fi

gitlog-to-changelog comes from gnulib.

If you choose to continue working with VC'd ChangeLog files,
consider using gnulib's git-merge-changelog.c.  That should
help alleviate the worst of your merge problems.

Jim