Re: need for flag for incompatible-changes

Re: need for flag for incompatible-changes

[Joseph H. Buehler]

Per Bothner <bothner@cygnus.com> writes:

> The problem:  We need to continue to make changes to gcc exception
> handling, mangling, etc.  Some of these will be incompatible changes.

Keep in mind when you make abi changes in the compiler, that some
things are distributed as binary-only (e.g. Motif) and thus *cannot*
be recompiled by end users.  Motif is undoubtedly C code, so perhaps
less of a problem, but there are undoubtedly a bunch of binary-only
C++ libraries out there too.

A vendor can just recompile and redistribute, you say?  I am
currently forced to compile Motif apps against the redhat 4.2 headers
and libraries, even though my system is 5.0, because the Motif that I
bought from redhat is not compatible with 5.0's X libraries (because
of glibc 2, not because of compiler changes).  I have not tried egcs
with this mess yet, but intend to, and will keep you posted.

Granted, though: the "e" in egcs stands for "experimental".

Joe Buehler

Re: need for flag for incompatible-changes

[Joe Buck]

> Keep in mind when you make abi changes in the compiler, that some
> things are distributed as binary-only (e.g. Motif) and thus *cannot*
> be recompiled by end users.  Motif is undoubtedly C code, so perhaps
> less of a problem, but there are undoubtedly a bunch of binary-only
> C++ libraries out there too.

Up until now, we've made no effort to keep binary-only C++ libraries
running.  We will have to break them again, at least one more time,
to implement namespaces.  We may want to attempt to figure out all
the things we may break and try to do them only once; if we don't
want a permanent fork that will have to be coordinated with gcc2.

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.com> writes:

 >> Keep in mind when you make abi changes in the compiler, that some
 >> things are distributed as binary-only (e.g. Motif) and thus
 >> *cannot* be recompiled by end users.  Motif is undoubtedly C code,
 >> so perhaps less of a problem, but there are undoubtedly a bunch of
 >> binary-only C++ libraries out there too.

 Joe> Up until now, we've made no effort to keep binary-only C++
 Joe> libraries running.  We will have to break them again, at least
 Joe> one more time, to implement namespaces.  We may want to attempt
 Joe> to figure out all the things we may break and try to do them
 Joe> only once; if we don't want a permanent fork that will have to
 Joe> be coordinated with gcc2.

If object files contained version numbers, you wouldn't have to break
anything.  (More precisely, you would only have to break things that
are older than the oldest version you want to continue supporting in
the linker.)

	paul

Re: need for flag for incompatible-changes

[Joe Buck]

> If object files contained version numbers, you wouldn't have to break
> anything.  (More precisely, you would only have to break things that
> are older than the oldest version you want to continue supporting in
> the linker.)

Where we have control over the linker (Linux, or platforms where GNU ld
will work), we could use the linker to handle some types of
incompatibilities.  In many cases, I'm afraid that all it will lets
us do is say "sorry, incompatible object files".

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Joe" == Joe Buck <jbuck@Synopsys.COM> writes:

 >> If object files contained version numbers, you wouldn't have to
 >> break anything.  (More precisely, you would only have to break
 >> things that are older than the oldest version you want to continue
 >> supporting in the linker.)

 Joe> Where we have control over the linker (Linux, or platforms where
 Joe> GNU ld will work), we could use the linker to handle some types
 Joe> of incompatibilities.  In many cases, I'm afraid that all it
 Joe> will lets us do is say "sorry, incompatible object files".

Good point, I was silently assuming GNU ld.  If version number is
encoded in a way that doesn't bother other linkers (e.g., by a magical
global symbol) then users of other linkers would be no worse off than
they would be otherwise.

I wonder about the comment that even GNU ld often would only be able
to say "sorry, incompatible object files".  Some things, like naming
(namespace support) clearly aren't like that.  Similarly, adding a
handler type to the exception table shouldn't be a problem either (the
linker can supply the implied default value when it sees an old
table).

It seems to me that the ability to create a backwards compatibility
mechanism should be a design goal, and a high priority one at that.
In other words, if a feature requires an object format change, it
should be done in such a way that the linker can continue to accept
both old and new formats and make the right things happen -- unless it
is really and truly impossible to do that.  (Are there examples where
that is so?)

Again, using network protocol design as a precedent here, there are
many cases where not having the goal caused incompatible protocols to
be designed, but I can't think of one where, once the goal was put in
place, it was found to be impossible to meet.  Yes, it does make
things harder, though usually not by much.  But the benefit obtained
is very large.  Flag days are *bad news*...

	paul

Re: need for flag for incompatible-changes

[Andi Kleen]

Joe Buck <jbuck@synopsys.com> writes:

> > If object files contained version numbers, you wouldn't have to break
> > anything.  (More precisely, you would only have to break things that
> > are older than the oldest version you want to continue supporting in
> > the linker.)
> 
> Where we have control over the linker (Linux, or platforms where GNU ld
> will work), we could use the linker to handle some types of
> incompatibilities.  In many cases, I'm afraid that all it will lets
> us do is say "sorry, incompatible object files".

I'm not sure if it is a good idea to let the linker know about intime
details of the exception table format. Looks like a strong layering
violation for me.

The correct way to solve this IMHO is to use the ELF Symbol versioning
support in binutils 2.8. It was exactly designed to solve problems
like this. In the worst case object files will grow a bit because
they'll contain two copies of the exception table, but not the
resulting executable because the linker will select the correct one
using the matching symbols.

[Afaik the current exception table works using a section instead of 
symbols, but maybe the symbol versioning support can be extended for
section names too?]

-Andi

Re: need for flag for incompatible-changes

[Jeffrey A Law]

  In message < 199801281711.JAA25083@atrus.synopsys.com >you write:
  > 
  > > Keep in mind when you make abi changes in the compiler, that some
  > > things are distributed as binary-only (e.g. Motif) and thus *cannot*
  > > be recompiled by end users.  Motif is undoubtedly C code, so perhaps
  > > less of a problem, but there are undoubtedly a bunch of binary-only
  > > C++ libraries out there too.
  > 
  > Up until now, we've made no effort to keep binary-only C++ libraries
  > running.  We will have to break them again, at least one more time,
  > to implement namespaces.  We may want to attempt to figure out all
  > the things we may break and try to do them only once; if we don't
  > want a permanent fork that will have to be coordinated with gcc2.
I'd like to see just a single breakage if at all possible.  I don't
think there's any disagreement on that :-)

However, what I think Per is looking for is a flag which we can put
in the compiler which enables whatever pending incompatable changes
we've already implemented so that we can at least start testing them.

As to coordination with gcc2.  Good point.  Note that this involves
coordination on the release level as well as the development level!



jeff

Re: need for flag for incompatible-changes

[Joe Buck]

> I wonder about the comment that even GNU ld often would only be able
> to say "sorry, incompatible object files".  Some things, like naming
> (namespace support) clearly aren't like that.

We not only plan to add namespace support to the compiler, but we
then need to redo libstdc++ to put the whole standard library in the
std namespace.  That's what breaks compatibility, and it doesn't show
up at the level of object file version -- it is a change in the API.
Adding namespaces wouldn't break compatibility if namespaces are not
used.

The other major change that needs to be done to C++ is to redo iostreams
as specified by the draft standard (make iostream a template).  This
risks breaking libc compatibility on Linux if it is not done carefully.

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.com> writes:

 >> I wonder about the comment that even GNU ld often would only be
 >> able to say "sorry, incompatible object files".  Some things, like
 >> naming (namespace support) clearly aren't like that.

 Joe> We not only plan to add namespace support to the compiler, but
 Joe> we then need to redo libstdc++ to put the whole standard library
 Joe> in the std namespace.  That's what breaks compatibility, and it
 Joe> doesn't show up at the level of object file version -- it is a
 Joe> change in the API.  

Then maybe you need more version numbers.

Clearly, if the names of the functions change (due to namespaces) but
their calling conventions did not (same argument types etc.) then the
linker can map old names to new names if it is told to do so.

	paul

Re: need for flag for incompatible-changes

[Richard Henderson]

On Thu, Jan 29, 1998 at 09:24:01AM -0500, Paul Koning wrote:
> Clearly, if the names of the functions change (due to namespaces) but
> their calling conventions did not (same argument types etc.) then the
> linker can map old names to new names if it is told to do so.

You know, at some point it becomes much easier to just
force people (and companies) to rebuild the libraries.


r~

Re: need for flag for incompatible-changes

[Joe Buck]

On Thu, Jan 29, 1998 at 09:24:01AM -0500, Paul Koning wrote:
> > Clearly, if the names of the functions change (due to namespaces) but
> > their calling conventions did not (same argument types etc.) then the
> > linker can map old names to new names if it is told to do so.

rth writes:
> You know, at some point it becomes much easier to just
> force people (and companies) to rebuild the libraries.

I agree.  The only reason for some kind of name-munging trick would be
to support old executables that are available only in binary form.
That's a job that probably isn't worth doing by the egcs team.  If
a specific customer has a need for it, it might be a nice way for some
consultant to make a bit of money.

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Joe" == Joe Buck <jbuck@Synopsys.COM> writes:

 Joe> On Thu, Jan 29, 1998 at 09:24:01AM -0500, Paul Koning wrote:
 >> > Clearly, if the names of the functions change (due to
 >> namespaces) but > their calling conventions did not (same argument
 >> types etc.) then the > linker can map old names to new names if it
 >> is told to do so.

 Joe> rth writes:
 >> You know, at some point it becomes much easier to just force
 >> people (and companies) to rebuild the libraries.

 Joe> I agree.  The only reason for some kind of name-munging trick
 Joe> would be to support old executables that are available only in
 Joe> binary form.  That's a job that probably isn't worth doing by
 Joe> the egcs team.  If a specific customer has a need for it, it
 Joe> might be a nice way for some consultant to make a bit of money.

That sounds reasonable.  But for it to be possible, the compiler has
to supply the raw data, i.e., it has to emit the version data into the
object files.

	paul

Re: need for flag for incompatible-changes

[Joe Buck]

>  Joe> I agree.  The only reason for some kind of name-munging trick
>  Joe> would be to support old executables that are available only in
>  Joe> binary form.  That's a job that probably isn't worth doing by
>  Joe> the egcs team.  If a specific customer has a need for it, it
>  Joe> might be a nice way for some consultant to make a bit of money.
> 
> That sounds reasonable.  But for it to be possible, the compiler has
> to supply the raw data, i.e., it has to emit the version data into the
> object files.

But these changes I'm describing have nothing to do with the compiler,
really, and everything to do with the library source.  And we already
have a library version number.

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Joe" == Joe Buck <jbuck@Synopsys.COM> writes:

 Joe> I agree.  The only reason for some kind of name-munging trick
 Joe> would be to support old executables that are available only in
 Joe> binary form.  That's a job that probably isn't worth doing by
 Joe> the egcs team.  If a specific customer has a need for it, it
 Joe> might be a nice way for some consultant to make a bit of money.
 >>  That sounds reasonable.  But for it to be possible, the compiler
 >> has to supply the raw data, i.e., it has to emit the version data
 >> into the object files.

 Joe> But these changes I'm describing have nothing to do with the
 Joe> compiler, really, and everything to do with the library source.
 Joe> And we already have a library version number.

We must have drifted off-subject... What started the discussion was
Per's proposal to add a compiler switch.  For cases where that is
meaningful, a compiler-generated version number in the object file
should also be meaningful, right?

	paul

Re: need for flag for incompatible-changes

[Joe Buck]

> Clearly, if the names of the functions change (due to namespaces) but
> their calling conventions did not (same argument types etc.) then the
> linker can map old names to new names if it is told to do so.

Yes, this might work (again, on platforms where we have control of the
linker).  It's a bit more complicated than just putting everything in
the std namespace though.  The iostreams classes are supposed to become
templates, so that ostream is now a typedef for basic_ostream<char>
(you can also have streams of wide characters).  It is not a small
job.

Re: need for flag for incompatible-changes

[Fred Eisele]

Paul Koning wrote:

> >>>>> "Joe" == Joe Buck <jbuck@synopsys.com> writes:
>
>  >> I wonder about the comment that even GNU ld often would only be
>  >> able to say "sorry, incompatible object files".  Some things, like
>  >> naming (namespace support) clearly aren't like that.
>
>  Joe> We not only plan to add namespace support to the compiler, but
>  Joe> we then need to redo libstdc++ to put the whole standard library
>  Joe> in the std namespace.  That's what breaks compatibility, and it
>  Joe> doesn't show up at the level of object file version -- it is a
>  Joe> change in the API.
>
> Then maybe you need more version numbers.
>
> Clearly, if the names of the functions change (due to namespaces) but
> their calling conventions did not (same argument types etc.) then the
> linker can map old names to new names if it is told to do so.

Isn't this the kind of de(re)mangling that <collect> was invented for?

Re: need for flag for incompatible-changes

[Joe Buck]

> > Clearly, if the names of the functions change (due to namespaces) but
> > their calling conventions did not (same argument types etc.) then the
> > linker can map old names to new names if it is told to do so.
> 
> Isn't this the kind of de(re)mangling that <collect> was invented for?

collect does not remangle symbols.  It finds symbols that match a
pattern (file-scope ctor/dtor calls).  It pipes errors from the linker
through the demangler, and that's about it.

If someone wants to "sign up" for a task that benefits only those who
distribute binary-only libraries or executables, fine, but otherwise
I doubt if any of the developers will want to bother.

Re: need for flag for incompatible-changes

[Mike Stump]

If we have a flag, I think I prefer a more general solution where we
select a flag that specifies what abi we want, and then use that.
When we make incompatible changes, then we conditionalize the new code
on the new value for the flag, and use the old code (with the
disadvantages it brings), with the old value.

-fcompat-2.8.0 is default (for now), and generates code compatible
with 2.8.0, -fcompat-2.9.0 likewise.

I'm not sure if we should do this or not, though I have thought about
it on and off for a while now.

Another way, is to just have a #define NEW_EH and then switch it on at
some point in the future, and then remove it sometime later.

need for flag for incompatible-changes

[Per Bothner]

The problem:  We need to continue to make changes to gcc exception
handling, mangling, etc.  Some of these will be incompatible changes.
For example, the current exception table has three columns:  start_pc,
end_pc, and handler_pc.  It is highly desirable (especially for Java)
to add a fourth column: handler_type.  We cannot do that without
breaking compatibility.  So we will break binary compatibility, but we
want to bunch up many compatibility-breaking changes and do them
all at once, to reduce disruption.  But it is easier to make and
test many small incremental changes.  So I propose we add a flag
to turn on compatibility-breaking changes.  The default will be
to disable the changes - except perhaps for Java, which needs the
changes, and has no installed base.  That way people can test them,
plus people who make unrelated changes are less likely to make
patches that don't work with the development sources.  Then
when the time comes, we can migrate the changes to be always enabled.

We could use -fexperimental-exceptions to enable compatibility-breaking
eh-related changes.  Or we could use a more general flag -fexperimental
as a catch-all.

Perhaps "experimeental" does not quite cover what I intend.
How about -fno-compatibility?

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: need for flag for incompatible-changes

[Joe Buck]

> The problem:  We need to continue to make changes to gcc exception
> handling, mangling, etc.  Some of these will be incompatible changes.

> We could use -fexperimental-exceptions to enable compatibility-breaking
> eh-related changes.  Or we could use a more general flag -fexperimental
> as a catch-all.

But there will be problems if you do more than one round of this: first
-fexperimental-exceptions, then that becomes the standard, and then you
want to do a new round of experimental exceptions.

You need to give each incompatible feature a more specific name.  You
can't pollute the name space by taking "experimental exceptions" unless
you assume that no one else will ever make a different experiment.

> How about -fno-compatibility?

I think that that is even worse.  Many possible changes can break
compatibility.

In your case, the change you propose is to add a fourth field to the
exception table, so ideally the flag should suggest precisely that.
Good abbreviations are hard to come by:
-fexceptions-with-handler-type is a bit wordy.
We could name it after you: -fexceptions-bothner98 (this lets you
do a different scheme next year :-).

Re: need for flag for incompatible-changes

[Per Bothner]

Joe Buck <jbuck@Synopsys.COM> writes:
> But there will be problems if you do more than one round of this: first
> -fexperimental-exceptions, then that becomes the standard, and then you
> want to do a new round of experimental exceptions.

You misunderstand.  I am not proposing a flag to control a specific
set of changes.  I am proposing a flag to signal "I am not concerned
about binary compatibility."  It is flag only intended for developers.
Then at each "flag day" (sic) we migrate a whole bunch of changes
from being under control of this flag, to become the default.

> In your case, the change you propose is to add a fourth field to the
> exception table, so ideally the flag should suggest precisely that.

No, that is just *one* of the changes,  though it is one of the more
major and obviously incompatible ones.  I don't want a flag for each
change.

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: need for flag for incompatible-changes

[Joe Buck]

> Joe Buck <jbuck@Synopsys.COM> writes:
> > But there will be problems if you do more than one round of this: first
> > -fexperimental-exceptions, then that becomes the standard, and then you
> > want to do a new round of experimental exceptions.
> 
> You misunderstand.  I am not proposing a flag to control a specific
> set of changes.  I am proposing a flag to signal "I am not concerned
> about binary compatibility."  It is flag only intended for developers.
> Then at each "flag day" (sic) we migrate a whole bunch of changes
> from being under control of this flag, to become the default.

OK, I get what you want to do.  Unfortunately, this *still* has to be done
very carefully, or we restore the "Linux users need not apply to hack gcc"
state of affairs we had up until egcs came out (because the world breaks
if your libc and libstdc++ aren't compatible, and things like
vtable-thunks must be set the same for both).

> > In your case, the change you propose is to add a fourth field to the
> > exception table, so ideally the flag should suggest precisely that.
> 
> No, that is just *one* of the changes,  though it is one of the more
> major and obviously incompatible ones.  I don't want a flag for each
> change.

Even though it might be convenient to have one command line flag, it's
probably best to have multiple internal flags.  Otherwise you have more
work when it's decided that three out of your five new features are good
enough to be the default -- you don't have to go searching through the
code, you just set three of the five internal flags to "enabled".

Re: need for flag for incompatible-changes

[Paul Koning]

>>>>> "Per" == Per Bothner <bothner@cygnus.com> writes:

 Per> The problem: We need to continue to make changes to gcc
 Per> exception handling, mangling, etc.  Some of these will be
 Per> incompatible changes. ... So I propose we add a flag to turn on
 Per> compatibility-breaking changes.  The default will be to disable
 Per> the changes - except perhaps for Java, which needs the changes,
 Per> and has no installed base.

A standard way in network protocol design to deal with incompatible
changes is to stick a version number in.  That has a number of
benefits.  For one thing, a version number has more than two values,
unlike a flag.  (So you don't get into trouble when you have to do a
second change a year or two later.)

Another aspect of protocol version number handling is that
implementations generally support at least two versions: the current
one and the one preceding that (if there is one).  As a result, while
the change may be incompatible at the protocol level, it is NOT
imcompatible from the user point of view, because the system adjusts
according to the version number it sees.

I'd suggest an analogous approach.  Is there a "binary format version
number" anywhere?  (There should have been, of course.)  If not,
perhaps one could be created by defining a new global symbol, starting
at 1, the absence of that global implying version 0.

Then you can use this version number to adjust the behavior as needed.

If it has to be per object file (which seems to be the case with
things like exception handler tables) this may still work: the (new)
linker could use the object file version number to determine what
format exception table it contains (and, quite possibly, convert the
old format to the new while linking old object files).

If there are things that cannot be converted by the development tools,
you should *still* have a version number; that way there can be error
messages generated that point out the specific issues.  Without it you
end up with mysterious crashes.  (In the case of exception handling,
it would be particularly nasty: a mysterious crash that happens *only*
if you invoke an exception in the wrong place.)

	paul