backwards incompatibility

backwards incompatibility

[Dave Love]

It's quite a pain for supporting research systems that the gfortran
module format changes incompatibly with every (?) new version.  For
instance, you can't mix development packages built with the RHEL system
compiler with the more recent "devtoolset" compiler needed for hardware
support.  As far as I know, the proprietary compilers don't have the
same issue.  I can't find anything written about that, but I haven't
tried to understand the implementation.  Is there something written?
Otherwise, is it feasible to convert older module files to newer
formats, given that the ABI doesn't change, as I've seen in release
notes (and would hope)?  If so, where would one look in the code to see
about doing so?

Version 8 makes the situation worse by breaking run time compatibility,
not just build-time, with a different libgfortran shared library
version.  That already caused considerable disruption with Fedora
packaging after the switch to GCC 8.  With ELF, can't you use versioned
symbols to avoid a new soname, like libgcc and libgomp do?

Regardless of any annoyances, thanks for keeping a competitive free
compiler going!

Re: backwards incompatibility

[Steve Kargl]

On Mon, Mar 12, 2018 at 03:40:48PM +0000, Dave Love wrote:
> It's quite a pain for supporting research systems that the gfortran
> module format changes incompatibly with every (?) new version.  For
> instance, you can't mix development packages built with the RHEL system
> compiler with the more recent "devtoolset" compiler needed for hardware
> support.  As far as I know, the proprietary compilers don't have the
> same issue.  I can't find anything written about that, but I haven't
> tried to understand the implementation.  Is there something written?
> Otherwise, is it feasible to convert older module files to newer
> formats, given that the ABI doesn't change, as I've seen in release
> notes (and would hope)?  If so, where would one look in the code to see
> about doing so?
> 
> Version 8 makes the situation worse by breaking run time compatibility,
> not just build-time, with a different libgfortran shared library
> version.  That already caused considerable disruption with Fedora
> packaging after the switch to GCC 8.  With ELF, can't you use versioned
> symbols to avoid a new soname, like libgcc and libgomp do?
> 
> Regardless of any annoyances, thanks for keeping a competitive free
> compiler going!

Hi Dave,

While I do not speak for the other contributors, I can
assure you that we take breaking the library ABI quite
serious.  We maintain a list of changes that we know will
cause problems.  If a change occurs that breaks the ABI,
we try to merge as many changes from the list as possible
to minimize on-going breakage.  See

https://gcc.gnu.org/wiki/LibgfortranAbiCleanup

for those that have yet to be merged/fixed.  Unfortunately,
the addition of user defined derived type IO, which was a
much requested feature, required a change to the library
internals for the upcoming 8.0.

Maintaining the internals of *.mod files is even more difficult
as changes are sometimes required for bug fixes as well as when
new features are implemented (e.g., parameterized derived type).

-- 
Steve

Re: backwards incompatibility

[Dave Love]

Steve Kargl <sgk@troutmask.apl.washington.edu> writes:

> Hi Dave,
>
> While I do not speak for the other contributors, I can
> assure you that we take breaking the library ABI quite
> serious.  We maintain a list of changes that we know will
> cause problems.  If a change occurs that breaks the ABI,
> we try to merge as many changes from the list as possible
> to minimize on-going breakage.  See
>
> https://gcc.gnu.org/wiki/LibgfortranAbiCleanup
>
> for those that have yet to be merged/fixed.  Unfortunately,
> the addition of user defined derived type IO, which was a
> much requested feature, required a change to the library
> internals for the upcoming 8.0.

Yes, sure.  I'm interested in whether there's something fundamental that
prevents symbol versioning keeping backwards compatibility, or if it's
basically a question of effort.

There's a related question about what happens if you try to link code
from, say gfortran 6 and 8.  (I forgot or missed that there was a break
at 7 too.)  Is it expected to work, fail to link, or potentially fail
mysteriously at run time?

> Maintaining the internals of *.mod files is even more difficult
> as changes are sometimes required for bug fixes as well as when
> new features are implemented (e.g., parameterized derived type).

Similarly, I'm wondering whether it should at least be possible to write
a translator from an old version to a new one if someone worked on it,
or whether that can't work.

Thanks for any insight.

Re: backwards incompatibility

[Thomas Koenig]

Hi Dave,

> It's quite a pain for supporting research systems that the gfortran
> module format changes incompatibly with every (?) new version.

A lot of releases, yes. The problem is that we use a lot of positional
parameters in the files.

I have given a little thought of how to get around that by using some
sort of keyword-value scheme, which by definition are almost infinitely
extensible.

Another possibility would be to write out INTERFACE blocks as *.mod
files, which could then be re-read and re-parsed.

Just two problems: To a) get the design right, and b) get the
implementation right.  This would take a lot of effort, and the
manpower simplfy isn't there (and it hasn't figured high on the
list of priorities).

Of course, volunteers are always welcome.

> For
> instance, you can't mix development packages built with the RHEL system
> compiler with the more recent "devtoolset" compiler needed for hardware
> support.  As far as I know, the proprietary compilers don't have the
> same issue.  I can't find anything written about that, but I haven't
> tried to understand the implementation.  Is there something written?

It's in module.c, no documentation of gfortran's module file has
ever been done. Volunteers welcome...

> Version 8 makes the situation worse by breaking run time compatibility,
> not just build-time, with a different libgfortran shared library
> version.

This had to be done in order to fix long-standing bugs, and to allow
Fortran 2008 compliance.  We tried to stuff as much as possible into
this release, to avoid breaking binary compatibility for as many
releases as possible.

> That already caused considerable disruption with Fedora
> packaging after the switch to GCC 8.

What's the problem? Shipping two versions of shared libraries together
cannot be that hard, can it? OpenSUSE tumbleweed, which already
ships a pre-gcc8, manages this fine.

> With ELF, can't you use versioned
> symbols to avoid a new soname, like libgcc and libgomp do?

There are other systems which command equal attention - AIX, cygwin,
MacOS. Of course, volunteers are always welcome...

Regards

	Thomas

Re: backwards incompatibility

[Dave Love]

Thomas Koenig <tkoenig@netcologne.de> writes:

> Hi Dave,
>
>> It's quite a pain for supporting research systems that the gfortran
>> module format changes incompatibly with every (?) new version.
>
> A lot of releases, yes. The problem is that we use a lot of positional
> parameters in the files.
>
> I have given a little thought of how to get around that by using some
> sort of keyword-value scheme, which by definition are almost infinitely
> extensible.
>
> Another possibility would be to write out INTERFACE blocks as *.mod
> files, which could then be re-read and re-parsed.
>
> Just two problems: To a) get the design right, and b) get the
> implementation right.  This would take a lot of effort, and the
> manpower simplfy isn't there (and it hasn't figured high on the
> list of priorities).
>
> Of course, volunteers are always welcome.

The immediate interest is in whether it's possible to write translations
of old .mods to newer formats to solve the existing issues, but that
presumably only makes sense if you can reliably mix code compiled with
old and new compiler versions.  I'll see how far I can get with module.c
when I have a chance, thanks.

>> Version 8 makes the situation worse by breaking run time compatibility,
>> not just build-time, with a different libgfortran shared library
>> version.
>
> This had to be done in order to fix long-standing bugs, and to allow
> Fortran 2008 compliance.  We tried to stuff as much as possible into
> this release, to avoid breaking binary compatibility for as many
> releases as possible.

Sure, I understand the need for mods of some sort.

>> That already caused considerable disruption with Fedora
>> packaging after the switch to GCC 8.
>
> What's the problem? Shipping two versions of shared libraries together
> cannot be that hard, can it? OpenSUSE tumbleweed, which already
> ships a pre-gcc8, manages this fine.

Yes, I have libgfortran.3 and .4 in RHEL, but it's not clear I can
(reliably) link libraries which require both of them in the same binary.
I suppose I should at least pull out abidiff.

>> With ELF, can't you use versioned
>> symbols to avoid a new soname, like libgcc and libgomp do?
>
> There are other systems which command equal attention - AIX, cygwin,
> MacOS. Of course, volunteers are always welcome...

I'm surprised AIX commands equal attention to GNU/Linux, but you have
ELF versioning anyway.  It's just not clear how libgfortran is different
from libstdc++ or libgomp and whether there are considerations past the
library ABI.

I hope it doesn't look as if I'm just moaning.  I might be able to do
some work, and I have some (very old) history with GNU Fortran.  I was
asking because I'd guess just reading source isn't good enough to
understand.

Re: backwards incompatibility

[Jerry DeLisle]

On 03/13/2018 08:14 AM, Dave Love wrote:
> Thomas Koenig <tkoenig@netcologne.de> writes:
> 
>> Hi Dave,
>>
>>> It's quite a pain for supporting research systems that the gfortran
>>> module format changes incompatibly with every (?) new version.
>>
>> A lot of releases, yes. The problem is that we use a lot of positional
>> parameters in the files.
>>
>> I have given a little thought of how to get around that by using some
>> sort of keyword-value scheme, which by definition are almost infinitely
>> extensible.
>>
>> Another possibility would be to write out INTERFACE blocks as *.mod
>> files, which could then be re-read and re-parsed.
>>
>> Just two problems: To a) get the design right, and b) get the
>> implementation right.  This would take a lot of effort, and the
>> manpower simplfy isn't there (and it hasn't figured high on the
>> list of priorities).
>>
>> Of course, volunteers are always welcome.
> 
> The immediate interest is in whether it's possible to write translations
> of old .mods to newer formats to solve the existing issues, but that
> presumably only makes sense if you can reliably mix code compiled with
> old and new compiler versions.  I'll see how far I can get with module.c
> when I have a chance, thanks.
> 

It would be an order of magnitude easier and take less time if you just 
recompiled all of your sources.  I hope you use make or cmake or at 
least some build scripting and let it run overnight.  Or, does it take a 
week or two?

Regards,

Jerry

Re: backwards incompatibility

[Dave Love]

Jerry DeLisle <jvdelisle@charter.net> writes:

> It would be an order of magnitude easier and take less time if you
> just recompiled all of your sources.  I hope you use make or cmake or
> at least some build scripting and let it run overnight.  Or, does it
> take a week or two?
>
> Regards,
>
> Jerry

[I saw this late on gmane as I'm not subscribed to the list.]

This isn't necessarily about "my" sources (stuff on local systems).  I
go with the engineering trades-off in packaging and late binding (in
this context) rather than combinatorial rebuild-the-world and distribute
packaging.

I'm specifically interested in RHEL (EPEL) packages, and concerned by
the lack of information on ABI compatibility of, say, gfortran 4.8 (the
system version) and gfortran 6 (the latest with the same libgfortan,
distributed for RHEL-like systems and necessary for several things like
proper MPI support and avx512).  Can someone say if 6 is (in)compatible
with 4.8 apart from the module format, assuming the same language
features?