[ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[Andrew Schulman via Cygwin-announce]

The unison2.48 package has been updated in Cygwin. This new release,
version 2.48.4-2, has been rebuilt with OCaml 4.08.1, which makes it
compatible for syncing with Ubuntu Focal and the current Debian testing.

*Warning*: This update may be a breaking change, if you use unison2.48 to
synchronize with hosts that run other Linux distros. This is because two
Unison executables have to be built with the same version of OCaml in order
to be compatible. For discussion of this newly-discovered problem and how
we plan to deal with it in Cygwin, see OCaml Versions, below.

== Unison Versions in Cygwin

Unison is packaged for Cygwin as several independent packages, each
providing a version of Unison that is incompatible with the others. You
have to run compatible versions of Unison on the client and server, or
Unison will issue an error message and quit. Two versions of Unison are
compatible only if the first two numbers in their version strings are the
same. For example, all versions 2.48.* are compatible with each other, and
incompatible with versions 2.51.*. By installing one or more of the
packages listed above, you can run whichever version you need in order to
synchronize with your server.

The unison* packages install a convenience symlink from /usr/bin/unison to
one of the versioned unison executables, e.g. unison-2.48.exe.
alternatives(8) is used to manage the symlink. If the symlink is being
managed in "auto" mode, then it will automatically track the highest
numbered version of unison that you have installed. You can override this
by either changing the symlink manually with ln(1), or, preferably, by
using alternatives(8):

alternatives --display unison   shows you the installed versions of unison,
their priorities, which executable the symlink points to, and whether the
symlink is being managed in "auto" or "manual" mode.

alternatives --config unison   presents a menu from which you can choose
which version you want to use as the default.  This puts the symlink in
manual mode.

alternatives --auto unison   puts the symlink in automatic mode, where it
will always point to the highest numbered version of unison that's
installed on your host.

alternatives --set unison /usr/bin/unison-$V.exe   forces the symlink to
point to unison-$V.exe, and puts it in manual mode.

Please see alternatives(8) for more details.

== OCaml Versions

There is unfortunately another layer of incompatibility in Unison: Two
Unison executables are only compatible if they were built with the same
version of OCaml. If you use, say, Unison 2.48 on two different hosts, but
one was built with OCaml 4.08.1 and the other with OCaml 4.10.0, you'll get
the dreaded error message

  Fatal error: Fatal error during unmarshaling (input_value: ill-formed
  message), possibly because client and server have been compiled with 
  different versions of the OCaml compiler.

This is a bad feature, of OCaml apparently. It means that in order to be
sure of being able to sync with another host, we could potentially need a
separate Unison package for every combination of Unison and OCaml.

For now, Unison packages for Cygwin are being built with the OCaml version
that seems to make them most generally compatible with other Linux distros:

Package     OCaml version  Compatibility
----------  -------------  -------------
unison2.27  ?              ?
unison2.32  ?              ?
unison2.40  ?              ?
unison2.45  ?              ?
unison2.48  4.08.1         Debian bullseye (testing), Ubuntu Focal
unison2.49  ?              ?
unison2.51  ?              ?

Packages with "?" are older builds, where the OCaml version isn't currently
known.

If your distro uses a different combination of Unison and OCaml versions
that you need in order to sync, please let me know here. If it's widely
used, I might update one of the existing builds, or make a new Cygwin
package for it.

Future versions of Unison may get around the OCaml version incompatibility.
For more discussion of the problem, see
http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-August/001972.html.

Andrew E. Schulman

Re: [ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[Ken Brown]

On 8/31/2020 4:47 PM, Andrew Schulman via Cygwin-announce wrote:
> The unison2.48 package has been updated in Cygwin. This new release,
> version 2.48.4-2, has been rebuilt with OCaml 4.08.1, which makes it
> compatible for syncing with Ubuntu Focal and the current Debian testing.
> 
> *Warning*: This update may be a breaking change, if you use unison2.48 to
> synchronize with hosts that run other Linux distros. This is because two
> Unison executables have to be built with the same version of OCaml in order
> to be compatible. For discussion of this newly-discovered problem and how
> we plan to deal with it in Cygwin, see OCaml Versions, below.

[...]

> == OCaml Versions
> 
> There is unfortunately another layer of incompatibility in Unison: Two
> Unison executables are only compatible if they were built with the same
> version of OCaml. If you use, say, Unison 2.48 on two different hosts, but
> one was built with OCaml 4.08.1 and the other with OCaml 4.10.0, you'll get
> the dreaded error message
> 
>    Fatal error: Fatal error during unmarshaling (input_value: ill-formed
>    message), possibly because client and server have been compiled with
>    different versions of the OCaml compiler.
> 
> This is a bad feature, of OCaml apparently. It means that in order to be
> sure of being able to sync with another host, we could potentially need a
> separate Unison package for every combination of Unison and OCaml.
> 
> For now, Unison packages for Cygwin are being built with the OCaml version
> that seems to make them most generally compatible with other Linux distros:
> 
> Package     OCaml version  Compatibility
> ----------  -------------  -------------
> unison2.27  ?              ?
> unison2.32  ?              ?
> unison2.40  ?              ?
> unison2.45  ?              ?
> unison2.48  4.08.1         Debian bullseye (testing), Ubuntu Focal
> unison2.49  ?              ?
> unison2.51  ?              ?
> 
> Packages with "?" are older builds, where the OCaml version isn't currently
> known.
> 
> If your distro uses a different combination of Unison and OCaml versions
> that you need in order to sync, please let me know here. If it's widely
> used, I might update one of the existing builds, or make a new Cygwin
> package for it.

What a mess!  Would you consider embedding the OCaml version in the package name 
as suggested by the Debian maintainer 
(http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-August/001975.html)? 
Thus unison2.48-2.48.4-1 would be replaced by something like 
unison2.48+4.04.1... and unison2.48-2.48.4-2 would be replace by something like 
unison2.48+4.08.1....  That way people who are happy with unison2.48-2.48.4-1 
can keep using it without being pestered by setup to update.

FWIW, here's my situation, which is probably not typical.  When you released 
unison2.48-2.48.4-1 a few years ago, it was incompatible with the Linux server 
that I sync with.  So I simply built OCaml and unison myself on that server, 
using the same versions that you used, and installed unison in my own ~/bin 
directory.

I could do that again, now using OCaml 4.08.1.  But I'd much rather just keep my 
current arrangement, without having setup try to update unison2.48 every time I 
run it.

Ken

Re: [ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[Andrew Schulman]

> > There is unfortunately another layer of incompatibility in Unison: Two
> > Unison executables are only compatible if they were built with the same
> > version of OCaml. 
> 
> What a mess! 

Glad you understand :)

> Would you consider embedding the OCaml version in the package name 
> as suggested by the Debian maintainer 
> (http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-August/001975.html)? 
> Thus unison2.48-2.48.4-1 would be replaced by something like 
> unison2.48+4.04.1... and unison2.48-2.48.4-2 would be replace by something like 
> unison2.48+4.08.1....  That way people who are happy with unison2.48-2.48.4-1 
> can keep using it without being pestered by setup to update.

Yes, I could do that. I was hoping to avoid it, but it may be the only realistic
solution. 

It may be mainly Unison 2.48 that's affected, and maybe also 2.51 and the
forthcoming 2.52. I guess that very few people are still using the older
versions, and if they are they don't want any changes now.
 
> FWIW, here's my situation, which is probably not typical.  When you released 
> unison2.48-2.48.4-1 a few years ago, it was incompatible with the Linux server 
> that I sync with.  So I simply built OCaml and unison myself on that server, 
> using the same versions that you used, and installed unison in my own ~/bin 
> directory.
> 
> I could do that again, now using OCaml 4.08.1.  But I'd much rather just keep my 
> current arrangement, without having setup try to update unison2.48 every time I 
> run it.

Thanks, that's helpful to know. Your situation might not be unusual, I don't
know. For all I know, you and I are the only Cygwin Unison users.

So was unison2.48-2.48.4-1 built with OCaml 4.04.1? I don't remember, and I'm
not sure how to tell.

Andrew

RE: [ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[David Allsopp]

Andrew Schulman wrote:
> > > There is unfortunately another layer of incompatibility in Unison:
> > > Two Unison executables are only compatible if they were built with
> > > the same version of OCaml.
> >
> > What a mess!
> 
> Glad you understand :)
> 
> > Would you consider embedding the OCaml version in the package name as
> > suggested by the Debian maintainer
> > (http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-
> August/001975.html)?
> > Thus unison2.48-2.48.4-1 would be replaced by something like
> > unison2.48+4.04.1... and unison2.48-2.48.4-2 would be replace by
> > something like unison2.48+4.08.1....  That way people who are happy
> > with unison2.48-2.48.4-1 can keep using it without being pestered by
> setup to update.
> 
> Yes, I could do that. I was hoping to avoid it, but it may be the only
> realistic solution.
> 
> It may be mainly Unison 2.48 that's affected, and maybe also 2.51 and the
> forthcoming 2.52. I guess that very few people are still using the older
> versions, and if they are they don't want any changes now.
> 
> > FWIW, here's my situation, which is probably not typical.  When you
> > released
> > unison2.48-2.48.4-1 a few years ago, it was incompatible with the
> > Linux server that I sync with.  So I simply built OCaml and unison
> > myself on that server, using the same versions that you used, and
> > installed unison in my own ~/bin directory.
> >
> > I could do that again, now using OCaml 4.08.1.  But I'd much rather
> > just keep my current arrangement, without having setup try to update
> > unison2.48 every time I run it.

Just for reference, the format of OCaml marshalled values changed in OCaml 4.08 (https://github.com/ocaml/ocaml/pull/1683). It's changed again in OCaml 4.11 as well (https://github.com/ocaml/ocaml/pull/8791) but in that case the new things which OCaml 4.11 can marshal would have caused errors in earlier versions, so I don't think that will affect Unison.

Quoting https://caml.inria.fr/pub/docs/manual-ocaml/libref/Marshal.html (emphasis mine): "The format for the byte sequences is compatible across all machines **for a given version of OCaml**" so it's an interesting design choice of Unison's.

That said, we (OCaml devs) change the marshal format as infrequently as possible, and go to considerable lengths to ensure that the older format can be read by newer runtimes: IIRC that particular change was the first break since OCaml 4.01 (Sep 2013).


David

Re: [ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[Ken Brown]

On 8/31/2020 8:39 PM, Andrew Schulman via Cygwin wrote:
>>> There is unfortunately another layer of incompatibility in Unison: Two
>>> Unison executables are only compatible if they were built with the same
>>> version of OCaml.
>>
>> What a mess!
> 
> Glad you understand :)
> 
>> Would you consider embedding the OCaml version in the package name
>> as suggested by the Debian maintainer
>> (http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-August/001975.html)?
>> Thus unison2.48-2.48.4-1 would be replaced by something like
>> unison2.48+4.04.1... and unison2.48-2.48.4-2 would be replace by something like
>> unison2.48+4.08.1....  That way people who are happy with unison2.48-2.48.4-1
>> can keep using it without being pestered by setup to update.
> 
> Yes, I could do that. I was hoping to avoid it, but it may be the only realistic
> solution.
> 
> It may be mainly Unison 2.48 that's affected, and maybe also 2.51 and the
> forthcoming 2.52. I guess that very few people are still using the older
> versions, and if they are they don't want any changes now.
>   
>> FWIW, here's my situation, which is probably not typical.  When you released
>> unison2.48-2.48.4-1 a few years ago, it was incompatible with the Linux server
>> that I sync with.  So I simply built OCaml and unison myself on that server,
>> using the same versions that you used, and installed unison in my own ~/bin
>> directory.
>>
>> I could do that again, now using OCaml 4.08.1.  But I'd much rather just keep my
>> current arrangement, without having setup try to update unison2.48 every time I
>> run it.
> 
> Thanks, that's helpful to know. Your situation might not be unusual, I don't
> know. For all I know, you and I are the only Cygwin Unison users.
> 
> So was unison2.48-2.48.4-1 built with OCaml 4.04.1? I don't remember, and I'm
> not sure how to tell.

I just checked a Cygwin snapshot from around the time unison2.48-2.48.4-1 was 
built 
(http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2017/02/22/131407/index.html), 
and it looks like the current version of Cygwin's OCaml was 4.02.3 at the time. 
I mentioned 4.04.1 above because that's the version I built for Linux.  So maybe 
the crucial thing is that it was pre-4.08.1.

Ken