Re: Linux vs. libio

Re: Linux vs. libio

[Mike Stump]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 16:38:54 -0800

> This isn't like old-IA32 vs. new-IA32, or old-fixinclude
> vs. new-fixincludes; there will never be a cutover from the old ABI
> to the new ABI.

?

> Instead, they'll both coexist, with a switch to choose which one to
> use, for the forseeable future.

Ick!  Jason!  Is this what you intended with this flag?  I never
attached the above meanings to this flag.

We normally would just put the ABI breaking chainges straight in, and
screw compatibility.  At some point, jason thought it would be good to
slow down breaking the ABI so he conditionalized the code so that it
could be tested, but that at release time (when we bought into
breaking the ABI), all the code conditional on the flag would enmass
become the default, and then we'd start all over and have a new set of
growing changes for the next ABI change.

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210101.RAA01743@kankakee.wrs.com >you write:
  > > From: Mark Mitchell <mark@codesourcery.com>
  > > Date: Mon, 20 Dec 1999 16:38:54 -0800
  > 
  > > This isn't like old-IA32 vs. new-IA32, or old-fixinclude
  > > vs. new-fixincludes; there will never be a cutover from the old ABI
  > > to the new ABI.
  > 
  > ?
Agreed.   My understanding was that the ia32 (and in fact every port) would
be converting to the new abi.  It doesn't make any sense to have multiple
mangling mechanisms, multiple EH mechanisms and multiple vtable layouts.

jeff

Re: Linux vs. libio

[Mike Stump]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 16:38:54 -0800

> This isn't like old-IA32 vs. new-IA32, or old-fixinclude
> vs. new-fixincludes; there will never be a cutover from the old ABI
> to the new ABI.

?

> Instead, they'll both coexist, with a switch to choose which one to
> use, for the forseeable future.

Ick!  Jason!  Is this what you intended with this flag?  I never
attached the above meanings to this flag.

We normally would just put the ABI breaking chainges straight in, and
screw compatibility.  At some point, jason thought it would be good to
slow down breaking the ABI so he conditionalized the code so that it
could be tested, but that at release time (when we bought into
breaking the ABI), all the code conditional on the flag would enmass
become the default, and then we'd start all over and have a new set of
growing changes for the next ABI change.

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210101.RAA01743@kankakee.wrs.com >you write:
  > > From: Mark Mitchell <mark@codesourcery.com>
  > > Date: Mon, 20 Dec 1999 16:38:54 -0800
  > 
  > > This isn't like old-IA32 vs. new-IA32, or old-fixinclude
  > > vs. new-fixincludes; there will never be a cutover from the old ABI
  > > to the new ABI.
  > 
  > ?
Agreed.   My understanding was that the ia32 (and in fact every port) would
be converting to the new abi.  It doesn't make any sense to have multiple
mangling mechanisms, multiple EH mechanisms and multiple vtable layouts.

jeff

Re: Linux vs. libio

[Jonathan Larmour]

Jeffrey A Law wrote:
> 
>   >  or (b) is marked
>   > very clearly as a local/temporary kludge.
> This is the kind of stuff that belongs in a local tree, not the GCC sources.

Or a branch? That would keep it out of the reach of normal users using the
CVS tree, even with -fnew-abi, while still permitting collaboration - which
is presumably the reason Mark wants it checked in to the gcc repository.

It's a bit more effort to merge the branch back in, but it can be done
discretely, without breaking anything.

Jifl

Re: Linux vs. libio

[Joe Buck]

I wrote:

> > Please note that I'm only one SC member.  If other members object to
> > what I'm saying above, we'd have to resolve it.

Per Bothner (another SC member) writes:

> I do object, or rather:  we should be very careful.  I think checking
> in changes for code that we do not "own" can very easily lead to
> divergence.

It's not as much of a difference as it might appear, since I'd want
things re-sync'd before releases, and I expect everyone with blanket
write privileges to "be very careful".

> In a pinch we can put in a temporary kludge, but only if it is
> clearly marked as such.

This may be an example of such a pinch.  The new ABI has to be tested.
Requiring all testers to rebuild their glibc means that there will be
a lot less testing, meaning a lower-quality release.

Re: Linux vs. libio

[Joe Buck]

I wrote:

> Are you sure that you understand Mark's proposal?  I ask this because
> your assertion is incorrect -- there would be two separate libio's
> involved.  (This is not a new thing: we used tricks like these back
> in the libc5 days sometimes).

Whoops, it appears that I was assuming that Mark was thinking of doing
things this way, but now I see that this is not at all clear.  But yes,
libio as originally written was designed to be used this way (either
as both the C and the C++ I/O implementation, or as only the C++
implementation).

Re: Linux vs. libio

[Jeffrey A Law]

  In message < m24sddz7ak.fsf@magnus.bothner.com >you write:
  > I haven't suggested that.  What I said is that no changes should be
  > checked into areas of the gcc cvs tree that we don't own (such as
  > texinfo and any libc parts of libio) unless either (a) it has first
  > been checked into the "upstream" master source tree,
That is, in effect, what I already stated.  Thus my stand that the libc
folks have to be intimately involved in this decision.  It is not a
decision that GCC folks should make on their own.

  >  or (b) is marked
  > very clearly as a local/temporary kludge.
This is the kind of stuff that belongs in a local tree, not the GCC sources.


[ ... ]

  > This is similar to the policy Cygnus has (had) for gcc: Nothing should
  > be in the Cygnus internal cvs tree unless it is also in the external
  > tree, or marked with special markers in the code.
Correct.  Note that Cygnus doesn't tend to push those local/temporary 
kludges into the official GCC sources, nor should anyone else.  Those
kludges are kept as local changes.


jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "H" == H J Lu <hjl@lucon.org> writes:

    H> Take a look at how libc5 is handled in libio in gcc. libio in
    H> gcc 2.95.2 is quite different from libio in libc5.

Thanks for your suggestion.  I'll see if that helps.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912202336.PAA16416@atrus.synopsys.com >you write:
  > 
  > >   > It isn't really required that libio and glibc change in lockstep.  It
  > >   > certainly doesn't matter on platforms that don't use glibc.
  > 
  > > Huh?  If you don't change them in lockstep you introduce incompatibilitie
  > s
  > > between the version in the linux C library and the version in GCC.  That
  > > has proven to be a majorly stupid thing to do.  We don't want to repeat
  > > that mistake again.  So I'll repeat.  They must change in lock step.
  > 
  > If glibc and libstdc++ are to share the same structures for streams/stdio,
  > then yes, they must change in lock-step.  This is appropriate for
  > *released* versions.
And do you happen to know when the next release cycle is going to start?  Are
you going to volunteer to remove this code if the release cycle starts before
glibc & gcc have merged their libio implementations?  Are you going to
volunteer to merge the two versions if Mark & CodeSourcery ultimately do not
do the work?

IMHO, even for development versions Mark needs to get the glibc folks on board
first or that work needs to happen on a branch.


  > But for *testing*, it would be nice to have the
  > *option* on Linux of saying "don't share i/o structures".  Enabling this
  > option would break programs that rely on the C++ standard's guarantee that
  > cout/stdout and cerr/stderr are synchronized (unless flush calls are
  > inserted at appropriate points), and we would never build binary packages
  > to be distributed in that way, though we could point users to this choice
  > if they are stuck, trying to get new C++ code to run on a legacy Linux
  > system, for instance.
All this can be easily done on a branch.

  > Ideally, this option would not represent a fork because we could find
  > an agreed-apon way to do this (agreeable to both the C++ and glibc teams).
Yup.  If we can do this, then all my objections are removed.  Ultimately this
is what I want -- Mark, CodeSourcery, Uli & the glibc team to agree on
a technical plan and work together to make it happen and keep the glibc
and gcc sources in sync.  That *must* happen eventually anyway, or everything
Mark & CodeSourcery are doing will end up being tossed into the trash can.

  > It appears that the only way to get past this impasse is to find a libio
  > patch that the glibc folks would be willing to accept.
Right.  That's precisely what needs to happen short and long term.  It's
the right thing to do.  While they are working towards such a solution Mark
and CodeSourcery are free to work on a branch to expose the code to additional
testers.

jeff

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

> >   o I'm not proposing changing *anything* about libio when compiled
> >     with the existing C++ ABI, which will continue to be supported
> >     indefinitely.
> > 
> >   o I'm proposing chaning libio, conditionally, when compiled with the
> >     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
> >     so, it will not work.

Geoff Keating writes:
> This is what I'm objecting to.  Perhaps I should draw a diagram.  I
> expect that with -fnew-abi, the version number of libstdc++ will
> change, perhaps to libstdc++.so.3.  Now, suppose I use my brand new
> GCC 3.0.0, with -fnew-abi on by default, under my linux system,
> to compile an executable 'test', and 'netscape' is an old binary-only
> application compiled under the old GCC.
> 
> So we have a shared library use graph that looks like this:
> 
> netscape --> libstdc++.so.2.8 --> libc.so.6
> test	 --> libstdc++.so.3   ----^
> 
> But this won't work with your proposal. 

Are you sure that you understand Mark's proposal?  I ask this because
your assertion is incorrect -- there would be two separate libio's
involved.  (This is not a new thing: we used tricks like these back
in the libc5 days sometimes).

His conditionally changed libio would live inside libstdc++.so.3, and the
C bindings for it would not be exported (exactly as it does on, say,
Solaris platforms).  Suppose test does some C stdio stuff in addition to
the C++ iostreams stuff.  In that case, it would get the iostreams stuff
from libstdc++.so.3, and would get the stdio stuff from libc.so.6.  There
would be two copies of libio in the same address space, one exporting the
C++ bindings and one exporting the C bindings.  The main user-visible
symptom would be that programs like this:

#include <stdio.h>
#include <iostream.h>

main ()
{
	printf ("This should be line 1.\n");
	cout << "This should be line 2.\n";
}

might not work as expected, because stdout and cout will not be
synchronized.  But most things will work just fine.  It would be much
the same as the situation on an OS with a proprietary libc: the
C++ and C i/o code would be separate.

Now, we don't want this as a long-term solution.  But it doesn't have
severe consequences as a temporary solution to be used by developers
and testers.

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> I wrote:

    >> Are you sure that you understand Mark's proposal?  I ask this
    >> because your assertion is incorrect -- there would be two
    >> separate libio's involved.  (This is not a new thing: we used
    >> tricks like these back in the libc5 days sometimes).

    Joe> Whoops, it appears that I was assuming that Mark was thinking
    Joe> of doing things this way, but now I see that this is not at
    Joe> all clear.  But yes, libio as originally written was designed
    Joe> to be used this way (either as both the C and the C++ I/O
    Joe> implementation, or as only the C++ implementation).

That was indeed my original intent.  The waters have become muddied at
this point, no doubt.

But, I think you have faithfully transcribed my original proposal, and
the one with which I feel the most comfort.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220110443J.mitchell@codesourcery.com >you write:
  > We're not talking about a release here.  We're just talking bout
  > development.
We do not change libio in our development tree without consulting with
and getting agreement from Ulrich and the glibc folks.  That is my
position and I see absolutely no reason to change it.

If you get them to buy into your scheme, then I'll support the code, but
that needs to happen *BEFORE* you install the code into GCC's libio.

  > It isn't really required that libio and glibc change in lockstep.  It
  > certainly doesn't matter on platforms that don't use glibc.
Huh?  If you don't change them in lockstep you introduce incompatibilities
between the version in the linux C library and the version in GCC.  That
has proven to be a majorly stupid thing to do.  We don't want to repeat
that mistake again.  So I'll repeat.  They must change in lock step.


  > It's very hard to move in lock-step; I think there are IA32 Linux
  > users who would like to try out -fnew-abi (to see if it works, to see
  > if it performs better, etc.), but who don't want to upgrade their C
  > libraries.  (I'm one of those users.)  
Yes it is hard; I've been in this situation many times.  Find a way to make
it work.  Find a way that is palatable to the glibc maintainers.

  > Why should we make it hard for people to test the new ABI?
Why should we diverge from glibc?  That just makes our life more difficult.

You've got find another way which doesn't cause our libio to diverge from
the one in glibc.  No ifs ands or buts about it.


jeff

Re: Linux vs. libio

[Martin v. Loewis]

> I think this is rather unwise, and ultimately the wrong approach to solving
> whatever problem you're working on.

It seems there is some confusion about the scope of what Mark attempts
to do.

He is working, in private, on implementing the new C++ ABI. Because he
is lacking an IA64 machine, he uses his private ia32 machine to do so.
To get it working at all, he must change libio, and he wants to avoid
rebuilding his Linux distribution to do so. I cannot find anything
unwise here.

Now he asks libio experts on how he should implement these changes, on
a technical level. All he gets as a response is "no, you don't want to
do that" (except for HJ Lu, who was giving some actual advice).

Please, can we leave the decision what to work on to the individual
developer, and only judge the quality of the work when we see actual
patches?

Having an independent copy of libio is only a stop-gap. We can look
for ways to integrate both copies into a single one later. That may or
may not be feasible. IMHO, changing the class layout is still the
right thing to do, and libio versioning should not stop us.

Regards,
Martin

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> Fortunately, the changes are relatively minimal: almost no code
    >> -- just a few changes in structure definitions and in a few
    >> macros.

    Ulrich> Adding these changes will probably be ok for other
    Ulrich> platforms (since there is no backward compatibility to be
    Ulrich> maintained) and it is also ok *for test purposes* to use
    Ulrich> this changed code on Linux, but no production release
    Ulrich> should be made with these changes and people should be
    Ulrich> discouraged to distribute libstdc++ versions compiled in
    Ulrich> this way.

Now I feel like you've heard what I was saying.

That's *exactly* what the purpose of the changes is -- other platforms
and testing.  That's what I said all along.

I can also imagine that we might have a configuration using the
un-integrated versino for people who have "old" libcs (like RH 6.x)
but want to use -fnew-abi.

Obviously, we're not going to ship anything with -fnew-abi until it's
stable, tested, etc.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220152927Y.mitchell@codesourcery.com >you write:
  > If that's what the steering committee, or whoever decides this kind of
  > thing, wants to do, then that's what we'll do.
A branch is certainly the direction I'm leaning.

  > But, I think it's really not a good idea.
  > 
  > In practice, this will lead to a lot less testing.
But it will avoid diverging our libio sources unnecessarily, which reduces our
development/support burden.

You have no idea (nor does anyone else right now) when the next release cycle
will start.  We don't need the extra burden of forcing the release manager to
pull your code out of the mainline sources because it wasn't finished, or the
glibc folks decided to go a different direction, etc etc.

Until such time as you've got a buy-in from the glibc folks you need to either
keep the code private or work from a branch.

  > Doing the new ABI work has already spotted bugs in the compiler, and
  > requires various other cleanups.  We'll lose those advantages in the
  > mainline, and there will be semi-serious divergence.
Just like the new ia32 port, garbage collection and new fixinclues did.  But
ultimately doing it on a branch was the right thing to do in each of those
cases.  I believe we're in a similar situation now -- from a maintenance
standpoint (as opposed to a stability standpoint).


  > We (CodeSourcery) might not have the resources to do the merge which
  > might mean the community may be unable to profit from the new ABI
  > until someone volunteers to do it.
That's the chance you and everyone must take when they decide to work with
free software projects.  If you don't have the resources to do the merge, then
you (in reality) under-bid the work by under-estimating the integration 
aspects of the contract.  I problem I am unfortunately all too familiar with.


  >   Think how long the GC work done on
  > the branch by Bernd and Richard might have languished had we not
  > stepped in to integrate it.
Yes.  I'm well aware of that.  Again, that's the chance you have to take to
play the free software game.  Note that the GC work was done entirely without
a customer and thus had nothing to drive it to completion.


  >   o We're not going to break IA32 Linux.
This isn't the issue.

  >   o We're not going to do stuff willy-nilly without asking the libio
  >     folks for approval.
This is the issue.  And I've stated before that if you get a buy-in from
the glibc folks then you can go forward.  But you need a buy in *before*
you start checking in the changes.

  > What's happenning here is that we're doing open development.
And you can just as easily do open development on a branch.  You can even
use the branch for highly experimental work to get it to a point where you
and the glibc folks agree on the general direction.

jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> If glibc and libstdc++ are to share the same structures for
    Joe> streams/stdio, then yes, they must change in lock-step.  This
    Joe> is appropriate for *released* versions.  But for *testing*,
    Joe> it would be nice to have the *option* on Linux of saying
    Joe> "don't share i/o structures".  Enabling this option would
    Joe> break programs that rely on the C++ standard's guarantee that
    Joe> cout/stdout and cerr/stderr are synchronized (unless flush
    Joe> calls are inserted at appropriate points), and we would never
    Joe> build binary packages to be distributed in that way, though
    Joe> we could point users to this choice if they are stuck, trying
    Joe> to get new C++ code to run on a legacy Linux system, for
    Joe> instance.

Exactly!  I'm ecstatic that you intuited what I was trying to say!

    Joe> It appears that the only way to get past this impasse is to
    Joe> find a libio patch that the glibc folks would be willing to
    Joe> accept.

That's what we were working on when I asked for help yesterday.
Sadly, we haven't made much progress; I've been too busy in this email
debate! :-(

I guess the mistake was to try to do the design in this way.  Instead,
we can just complete the patch, and present the finished product for
comment, and, hopefully, approval.

Let's avoid branching, then.  How about this:

  o We'll work on -fnew-abi support in the compiler.  Checking in the
    changes may break -fnew-abi regression tests, because of libio
    problems, but people can still see the changes and test -fnew-abi
    in non I/O streams examples.

  o In parallel, we'll work on the libio patches.  We'll submit them,
    and try to get approval.  Then, we'll put them in libio, bringing
    things back to zero regression-test failures.

There's no risk: nobody is supposed to be using -fnew-abi for anything
serious right now, non-new-abi builds won't break, and we'll keep the
libios in sync.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> IMHO, even for development versions Mark needs to get the
    Jeffrey> glibc folks on board first or that work needs to happen
    Jeffrey> on a branch.

There are reasons to get the new ABI code in, even without libio
changes.  Such as anyone using GCC on an IA64 system needs those
changes.  They can get another free C++ iostreams library, like the
one from SGI, and use it instead of libio.  

But, they can't ship ABI-compliant code without an ABI-compliant C++
compiler.

Doing the new ABI work (in the compiler, not libio) is just like
adding support for the MIPS R10K, or for a new generic optimization,
or for a new language feature -- as long as it's not impacting the
rest of the compiler, and it's a good thing to do, it belongs on the
mainline, right?  This isn't like old-IA32 vs. new-IA32, or
old-fixinclude vs. new-fixincludes; there will never be a cutover from
the old ABI to the new ABI.  Instead, they'll both coexist, with a
switch to choose which one to use, for the forseeable future.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> This may be an example of such a pinch.  The new ABI has to be tested.
> Requiring all testers to rebuild their glibc means that there will be
> a lot less testing, meaning a lower-quality release.

I haven't suggested that.  What I said is that no changes should be
checked into areas of the gcc cvs tree that we don't own (such as
texinfo and any libc parts of libio) unless either (a) it has first
been checked into the "upstream" master source tree, or (b) is marked
very clearly as a local/temporary kludge.  The former is much to be
preferred.  I certainly don't want to require people to re-build
glibc, unless they have a glibc *snapshot* and *want to* re-build
everything.

This is similar to the policy Cygnus has (had) for gcc: Nothing should
be in the Cygnus internal cvs tree unless it is also in the external
tree, or marked with special markers in the code.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991219183255Q.mitchell@codesourcery.com >you write:
  >   In other words, what I'm looking to do is to have two distinct,
  > disentangled, copies of libio: one in libc, to handle stdio, and one
  > in libstdc++ to handle iostreams.  How can this be done?
I think this is rather unwise, and ultimately the wrong approach to solving
whatever problem you're working on.

While I believe it was an unbelievably dumb thing for the glibc folks to
tie libio into the C library, it's done and the GCC folks have to deal with
it.

You will need to work with the glibc folks to find a solution that makes sense
to them as we do not want to start changing the API of the libio contained
in GCC without making corresponding changes to the libio contained in glibc.


jeff

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220163854A.mitchell@codesourcery.com >you write:
  > >>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:
  > 
  >     Jeffrey> IMHO, even for development versions Mark needs to get the
  >     Jeffrey> glibc folks on board first or that work needs to happen
  >     Jeffrey> on a branch.
  > 
  > There are reasons to get the new ABI code in, even without libio
  > changes.  Such as anyone using GCC on an IA64 system needs those
  > changes.  They can get another free C++ iostreams library, like the
  > one from SGI, and use it instead of libio. 
Sorry.  IMHO, you must get the glibc folks on board before you can check in
these changes to libio.

If you want I'll ask for a formal vote from the steering committee.

 >  the new ABI work (in the compiler, not libio) is just like
  > adding support for the MIPS R10K, or for a new generic optimization,
  > or for a new language feature -- as long as it's not impacting the
  > rest of the compiler, and it's a good thing to do, it belongs on the
  > mainline, right?  This isn't like old-IA32 vs. new-IA32, or
  > old-fixinclude vs. new-fixincludes; there will never be a cutover from
  > the old ABI to the new ABI.  Instead, they'll both coexist, with a
  > switch to choose which one to use, for the forseeable future.
I've got no objection to the compiler side of the ABI work going into the
development tree.  My objections are with such changes going into libio
without prior approval from the glibc/libio folks.  I thought I had made
that clear.

jeff

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> Still, neither you nor I can answer my procedural question. :-) It's a
> steering committe issue: they (poor things!), not us, get to decide
> these questions. :-)

Not all code in egcs is governed by the steering committee.  Just
think about texinfo.  Simply because code was added to egcs for
convenience reasons does not make the SC the owner.  It can decide
about the future of that copy (= branch) but we all certainly want to
avoid branches.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> That is, in effect, what I already stated.  Thus my stand
    Jeffrey> that the libc folks have to be intimately involved in
    Jeffrey> this decision.  It is not a decision that GCC folks
    Jeffrey> should make on their own.

I don't believe I ever suggested making this decision on my own.  I
certainly did not mean to.

This thread *began* with me asking Ulrich for help, after all!

I *of course* intend to run any patches past the libio people.

    >> or (b) is marked very clearly as a local/temporary kludge.

    Jeffrey> This is the kind of stuff that belongs in a local tree,
    Jeffrey> not the GCC sources.

Why?  Without changes, nobody can test the new ABI.  And, there's no
risk to the current ABI.  So, the only thing that go wrong is that we
break the new ABI, but that's clearly not supported by GCC yet.

Are you suggesting I check in the new ABI changes, but *not* the libio
changes, thereby leaving things in a contradictory state?

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 18:06:29 -0800

>     Geoff> So we need a libio in glibc which will work for _both_ the
>     Geoff> old ABI and the new ABI.
> 
> If we can do this, that will be wonderful.  I'm in no way opposed to
> this, if it is technically feasible.  I'll try to see how this might
> be done.
...
> But, one of the vtables will contain thunks that adjust the this
> pointer before calling through the real vtable.  So there will be two
> costs to just this level of compatibility: extra space, and extra
> time.  That seems undesirable to me.

It's not like there is a _choice_.  We must maintain backwards
compatibility.  If this costs in space/time, then that's unfortunate
but we have to live with it (and try to minimize it, of course).

If we did have to do this, we could at least try to make the runtime
cost apply only to old applications.  That way, it will diminish over
time.

> Can you suggest a way of making this work that doesn't involve a
> performance hit?

Another way to make it work is to have glibc detect the version of gcc
in use and have the new libio be the default when the new gcc is being
used, through the use of symbol versioning---that is, have two
completely separate copies of libio inside glibc "like multilibs, but
at runtime" (now _there's_ a slogan that inspires confidence :-) ).
Possibly something could be done to make this work without recompiling
glibc.  You couldn't mix C++ shared objects compiled with both
compilers, but I don't think this is a serious problem.

One advantage of this is that we could make the new-ABI code more
efficient, as it would avoid some of the backwards compatibility
baggage we have now.

>     Geoff> more complicated if you only want one library.  Hopefully
>     Geoff> the new C++ symbol mangling and the old one don't overlap
>     Geoff> so the 'more complicated' just means specifying both
>     Geoff> versions of the object files on the link line.
> 
> The mangling has not yet been decided, so that's unclear.  But note
> that, in your proposal, even the C parts of libio need to handle both
> ABIs, so I don't think mangling is enough to keep things totally
> separate.

I'm not sure that libio needs to handle both ABIs _in the same
process_.  Just inside the same libc.so.

>     Geoff> libio is special because it is part of glibc.  The C++ ABI
>     Geoff> changes at pretty frequent intervals, the C ABI doesn't.
> 
> It is the intention of myself (and I believe Jason Merrill would
> agree, after the new ABI is implemented, never to change it again --
> just as you folks have committed to with libc.  We recognize that one
> of the obstacles to widespread C++ acceptance has been the lack of a
> stable ABI, and I'm hoping that the new ABI, which has been developed
> by a committee of intelligent people, will be good enough that we can
> stick with it indefinitely.  Time will tell, of course.

Oh good!  I would very much support this.

> One of my concerns is testing: I want people to be able to test the
> new ABI.  Lots of those potential testers are on IA32 Linux, but they
> don't really want to upgrade their systems, or even have to install
> new versions of libc in funny hidden places just to do the testing.

For testing purposes, you can just say "check libio out from this
branch" and put whatever hacks you like in it; that's no problem.
But before the release we do need to solve the whole problem.

> But I think you and I will have to agree to disagree on this point; I
> think we just evaluate the likelihood of people being willing to
> upgrade differently.



-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210049.QAA17857@atrus.synopsys.com >you write:
  > I wrote:
  > 
  > > If glibc and libstdc++ are to share the same structures for streams/stdio
  > ,
  > > then yes, they must change in lock-step.  This is appropriate for
  > > *released* versions.
  > 
  > Jeff writes:
  > 
  > > And do you happen to know when the next release cycle is going to start? 
  >  Are
  > > you going to volunteer to remove this code if the release cycle starts be
  > fore
  > > glibc & gcc have merged their libio implementations?  Are you going to
  > > volunteer to merge the two versions if Mark & CodeSourcery ultimately do 
  > not
  > > do the work?
  > 
  > Is this rhetorical or do you mean it?  In case you do mean it:
  > If that extremely unlikely event occurred, I'd just ship the thing and no
  > one would care.
Both.  And people would care -- every time we've mucked around with the
internals of libio we've created problems.  I will not support installing
this kind of code into the development libio without the glibc folks getting
on board first.  Even if the code is only conditionally used.

libio is (in effect) an external set of sources that we just happen to keep
a copy of.  We should not be mucking around with it.  Doing so just makes
our life harder when we have to untangle the mess that making private changes
to libio creates.  I will not be a party to such stupidity.

Even if we think we've got the code 100% correct (ie the behavior only changes
for -fnew-abi), then I still consider it unacceptable to change libio without
prior consent from the libio folks.  No users's won't notice, but developers
that have to apply bugfixes, or bring in a new version of libio from glibc
will be affected in a negative way.  Merges suck in a major way.  We've
managed to kill the last need for a merge when we picked up the remaining
gcc2 changes.  Let's not introduce new merge requirements.

  > Nevertheless, we're in violent agreement: the way to proceed is to start
  > out with a separate branch, then work on patches that all can accept.
Right.  So are you and I agreed, get the code onto a branch to help facilitate
development until Mark & Uli can come to an agreement on how to handle the
libio changes?

jeff

Re: Linux vs. libio

[Marc Lehmann]

On Mon, Dec 20, 1999 at 03:29:27PM -0800, Mark Mitchell <mark@codesourcery.com> wrote:
>     Jeffrey> Not a bad idea.  On a branch would be fine by me.  We
>     Jeffrey> have used that scheme for similar purposes in the past.
> 
> If that's what the steering committee, or whoever decides this kind of
> thing, wants to do, then that's what we'll do.
> 
> But, I think it's really not a good idea.
> 
> In practice, this will lead to a lot less testing.
> 
> Doing the new ABI work has already spotted bugs in the compiler, and
> requires various other cleanups.  We'll lose those advantages in the
> mainline, and there will be semi-serious divergence.

Frankly, I do not understand these arguments.

While I am in favour of including this work in cvs, the idea of a branch
seems very reasonable for me.

If you wanted, you could make -fnew-abi the default there, so, instead of

  "for testing, you compile with -fnew-abi"

you do this:

  "for testing, check out -rlibio_new_abi"

My main problems with your reasoning is that you say, on one side: "all
code will be #ifdef'd out", but on the other side: "a branch will not be
tested as much" and "we might even be able to detect errors".

I think this needs clarification. For me, none of your arguments against a
branch seem to have substance.

Many similar projects have started as a branch first, and are now
integrated in the mainline sources.

Doing it on a branch (but in cvs) is as much open development as it being
on the main trunk. And it is a *major* help for the other maintainers (and
for the release manager(s)), in that it isolates the (substancial) patch
from the rest.

I do think you could just as well live with a branch, at least for some
time, until a solution for the libio problem is found.

And we do appreciate your work!

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@opengroup.org |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> But, there's still a question remaining: are people with
    >> blanket write privs allowed to change libio/libstdc++ without
    >> your approval, or not?

    Ulrich> I certainly reserve all rights for the libio parts as I'm
    Ulrich> probably the only one who sees the whole picture (libio in
    Ulrich> libstdc++ and in glibc).  As for the rest of libstdc++,

I'm not going to tread on your turf.  Obviously, you feel very
strongly about this.  And, obviously, I'd consult you anyhow -- I
already did!

Still, neither you nor I can answer my procedural question. :-) It's a
steering committe issue: they (poor things!), not us, get to decide
these questions. :-)

    Ulrich> that there were enough tests in the first place).  Plus:
    Ulrich> the version in libstdc++ is not the latest one (i.e., not
    Ulrich> the development version).

But, it is the version we ship with the compiler.  So, it's the only
one we can easily update to work with the new ABI, at this point.
Obviously, at some point, we'll have to merge the changes -- that
happens anytime a subproject is shared between two others.

If you want to update the version of libio in the compiler, by all
means go ahead!

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

> I'd like reassurance on the following point: assuming there is some change
> in gcc that is necessary for the larger good of gcc (specifically, a modern
> C++ ABI, without vtables/thunks related bugs, and liable to work correctly
> from the start on IA64), that may mean some unpleasantness for glibc or
> linux folks (like having to upgrade the glibc version number...), will the
> Steering Committee take steps to ensure the Right Thing is done ?  

IMHO, the decision is up to the glibc people.  By that I mean that we've
already more or less committed that GCC 3.0 will by default support the
new all-singing all-dancing good-until-a-new-committee-breaks-things
C++ ABI.  The glibc maintainers can either preserve the same version
number by using whatever hooks are necessary, or upgrade the version
number.  If they can do the former without significant performance
penalties, then that will be the preferable choice.

BTW, how are the BSD folks handling the C++ requirements for consistency
between stdout/cout and stderr/cerr?

Re: Linux vs. libio

[Marc Espie]

In article < 199912210311.TAA11055@localhost.cygnus.com > you write:
>> From: Mark Mitchell <mark@codesourcery.com>
>> Date: Mon, 20 Dec 1999 18:06:29 -0800
>
>>     Geoff> So we need a libio in glibc which will work for _both_ the
>>     Geoff> old ABI and the new ABI.
>> 
>> If we can do this, that will be wonderful.  I'm in no way opposed to
>> this, if it is technically feasible.  I'll try to see how this might
>> be done.
>...
>> But, one of the vtables will contain thunks that adjust the this
>> pointer before calling through the real vtable.  So there will be two
>> costs to just this level of compatibility: extra space, and extra
>> time.  That seems undesirable to me.

>It's not like there is a _choice_.  We must maintain backwards
>compatibility.  If this costs in space/time, then that's unfortunate
>but we have to live with it (and try to minimize it, of course).


This makes for yet another good reason to avoid linux and glibc.

Rude ? Maybe.

I've followed the glibc issues from the sidelines.
Somehow, I don't quite believe that, no matter how well-meaning the glibc
folks have been, such disasters won't happen again...  


Actually, I'm concerned about gcc and the steering committee position here.
I can understand that other FSF projects have a somewhat important influence
on gcc's life.

*However*, not all the world is FSF, there are other major users of gcc
outside of the glibc/linux worlds.

I'd like reassurance on the following point: assuming there is some change
in gcc that is necessary for the larger good of gcc (specifically, a modern
C++ ABI, without vtables/thunks related bugs, and liable to work correctly
from the start on IA64), that may mean some unpleasantness for glibc or
linux folks (like having to upgrade the glibc version number...), will the
Steering Committee take steps to ensure the Right Thing is done ?  


Furthermore, if you believe you will be able to live forever with glibc having
the same major version number, I think you're going down a very dangerous
path.

Updates happen.  Binary compatibility is not forever.  Update paths should be
provided, but the show must go on, problems must be fixed, and sooner or
later, there is going to be no other possibility than break old stuff (or get
more bloat than Windows).  As far as I understand, symbol versioning will
get you bloated systems/bloated libraries... ultimately a bad thing.
Whom does this help ?
* people for whom the update is too difficult,
* companies that distribute binary-only applications.

Both problems have real solutions: give people better tools to manage updates,
so that it becomes realistic for them to do so (notice how Windows makes 
this automatic... even though I loathe windows), 
and *don't* cater to companies that need such fine-grained binary 
compatibility. They can either put updates out, or start playing open, 
and work in a more system-frienly way.

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> Please note that I'm only one SC member.  If other members object to
> what I'm saying above, we'd have to resolve it.

I do object, or rather:  we should be very careful.  I think checking
in changes for code that we do not "own" can very easily lead to
divergence.  It is all too easy to put in a fix in our sources, and
forget about it.  Unless it also gets into the master sources, we now
have a potential future maintainance/merge problem.

As a general rule, we should only check in changes to foreign
code if it has been approved by the "upstream" maintainer.
Ideally, we should only make changes that will appear in some
future official release of that code, preferably the next one!
In a pinch we can put in a temporary kludge, but only if it is
clearly marked as such.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Ulrich Drepper]

Kevin Atkinson <kevinatk@home.com> writes:

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

No, it's not slow.  What Per is talking about are functions like
fputc, fgetc, also fputs, fgets etc.  These work on buffers and
normally don't include file operations.  And so it is a big handicap
to have extra indirection.

And something which wasn't mentioned: this methods requires a lot of
porting to each new platform (just look at how many platforms SGI's
stuff is working) and, perhaps more important, it is very risky.  The
libc is free to change the internals of the FILE structure since it is
an opaque type.  In glibc 2.2, for example, there will be support for
wide characters.  And the relations between the different buffers in
this context is really complicated.

So, manipulating the pointers in the FILE struct is not only slower,
it's also far less portable.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Kevin Atkinson]

Per Bothner wrote:

> One can write a C++ wrapper class (say stdiobuf) that implements the
> C++ functionality, but uses the C stdio buffers.  The problem is that
> this is slow, since you have to call handle_slow_case on every
> character.  You can force that by setting read_pointer == read_limit
> (for those streams that are aliased to stdio).  Alternatively, you can
> make sure to use the same value of read_pointer for C FILEs and C++
> filebufs.  However, then either you get extra overhead (indirection)
> or you have to merge the data structures for C and C++.

From http://www.sgi.com/Technology/STL/standard_library.html:

Another place where we've got system dependent code is syncing with
stdio. The C++ standard requires that there be a mode in which
iostreams and stdio are synchronized (the C++ standard is imprecise
about just what synchronization means, but, after talking to the
people who wrote that part of the standard, it's clear that it means a
very close coupling), and it requires that synchronized mode be the
default. We could have satisfied this requirement by implementing
things in terms of putc and getc, but that would have been horribly
inefficient. Instead we did something uglier, but more efficient: we
directly manipulate the pointers inside a FILE object. We've
encapsulated knowledge of struct FILE into the header
<stl_stdio_file.h>, which defines a few inline functions for this
manipulation. Again, this header has to be modified for every OS that
we target. This has not proven to be a serious problem in practice,
since most stdio implementations appear to be very similar to each
other.

Which is what I was trying to get at.  By directly maniplulating FILE
one avoids the extra overhead.

-- 
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> >From http://www.sgi.com/Technology/STL/standard_library.html:
> ...
> Which is what I was trying to get at.  By directly maniplulating FILE
> one avoids the extra overhead.

Wise man say: "Every problem in computer science can be solved
by adding an extra level of indirection."   While SGI's technique
is clever, it is basically just another layer of indirection.
Specifically, you get at least two extra memory de-references
even in the fast case.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> Not a bad idea.  On a branch would be fine by me.  We
    Jeffrey> have used that scheme for similar purposes in the past.

If that's what the steering committee, or whoever decides this kind of
thing, wants to do, then that's what we'll do.

But, I think it's really not a good idea.

In practice, this will lead to a lot less testing.

Doing the new ABI work has already spotted bugs in the compiler, and
requires various other cleanups.  We'll lose those advantages in the
mainline, and there will be semi-serious divergence.

We (CodeSourcery) might not have the resources to do the merge which
might mean the community may be unable to profit from the new ABI
until someone volunteers to do it.  Think how long the GC work done on
the branch by Bernd and Richard might have languished had we not
stepped in to integrate it.  This work needs to be done and in
official GCC before IA64 Linux systems go into any kind of serious
use, or there are going to be major headaches down the road -- the
same kind of libc5/libc6 thing will happen all over again.

I think that's perhaps what is still not being understood is:

  o We're not going to break IA32 Linux.

  o We're not going to do stuff willy-nilly without asking the libio
    folks for approval.

  o The only possible breakage will be for -fnew-abi programs,
    which are already not supported -- the new ABI is clearly marked
    as experimental, not for production use, may change, etc.

Branches make sense where there is real risk, or where completion is
uncertain.  For example, the new IA32 back-end branch was an excellent
use of a branch -- had the work been done on the mainline, it probably
would have taken a long time before anything could have been checked
in.  Lots of people would have lost time debugging problems in the new
back-end.  Instead, Richard worked on a branch, interested parties
helped, and finally the result was merged, with minimal impact (except
that our programs run faster now!).  But, the new ABI doesn't affect
anybody -- you only get it if you use -fnew-abi!

What's happenning here is that we're doing open development.

We could keep all our changes locally, give them just to our
customers, prevent the net from helping with the testing and
development load, and prevent people from reaping the benefits of the
new ABI.  Doing them on a branch has many of those same disadvantages.

Up until now we've *never* done any GCC development anywhere but n the
100% open light of day.  We've checked things into GCC when they were
tested and working, and we've fixed problems that arose as quickly as
possible.  We haven't even waited for our customers to pay us for the
work.  We've checked in things as we did the work, allowing everyone
to comment, criticize, test, and improve our efforts.  I'd like to
continue that policy -- I think that's the spirit of open source
development.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Per Bothner]

Jeffrey A Law <law@cygnus.com> writes:

> While I believe it was an unbelievably dumb thing for the glibc folks to
> tie libio into the C library,

In other words, you don't think it is a reasonable goal to allow C
stdio FILEs to be shared with C++ iostream streambufs?  The best way
to synchronize (say) stdout with cout seems to be for them both to use
the same buffer.  The requires (at least if you want efficiency) for
them to use compatible data structures, and thus libio has to be tied
to the C library.  (Note: This is not new to glibc; it has been that
way in Linux for many years.)
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Geoff Keating]

Mark Mitchell <mark@codesourcery.com> writes:

> Folks --
> 
>   Does anyone know how, on Linux, to build the libio (in CVS with GCC)
> in such a way as to be entirely disentangled from the C library?
> (Except in the ways that libio would normally be entangled with the C
> library on any non-glibc system.)
> 
>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.

I think this is a sign that you cannot change libio like that.  These
routines are interfaces between glibc and libio.  Yes, I know, it's
evil, but it's no worse than libgcc which is the same (ever heard of
__register_frame_pointer?).

It may be that you will have to change glibc to be compatible with the
new layout.  You've already changed glibc to be compatible with the
new EH so you will be requiring a new version of glibc anyway.

>   In other words, what I'm looking to do is to have two distinct,
> disentangled, copies of libio: one in libc, to handle stdio, and one
> in libstdc++ to handle iostreams.  How can this be done?

You don't want to do that.  It would cause C stdio and C++ iostreams
to be buffered separately causing output to be interleaved wrongly.
It would also break the nice stdio/iostreams interface that took so
much work to get right.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Jamie Lokier]

Mark Mitchell wrote:
> With the new ABI, pointers to virtual tables come at a different point
> in the objects.  Bases are laid out in different orders.  There are no
> longer pointers to virtual bases.  Some of that doesn't matter; some
> of it does.  Given:
> 
>   struct _ios_fields { ... };
>   class ios : public _ios_fields { ... };
> 
> it is no longer the case that you can cast an `ios *' to an
> `_ios_fields *' without having to do modify the pointer value;

> More importantly, the same holds for streambuf and _IO_FILE.  You
> can't cast an _IO_FILE to a streambuf, or vice versa, without
> modifying the pointer value.

<evil>
You might be able to do it by reserving a field in the new structures at
the right place to contain the old-Glibc-compatible vtable pointer, and
having two separate vtables.  Only one vtable (and corresponding
structure layout) is active at a time.  The other points to a set of
thunks.  When you call a thunk, it modifies the vtable pointers to swap
which is the active vtable, reorders the data fields for the other ABI
and continues from there.  
</evil>

;-)

-- Jamie

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> main ()
> {
> 	printf ("This should be line 1.\n");
> 	cout << "This should be line 2.\n";
> }
> 
> might not work as expected, because stdout and cout will not be
> synchronized.  But most things will work just fine.  It would be much
> the same as the situation on an OS with a proprietary libc: the
> C++ and C i/o code would be separate.

This is required to work according to the C++ standard, so it better
work on an OS with a proprietary libc.  However, it will work much less
efficiently.

What will stop working is:

        fprintf((FILE*) cout.rdbuf(), "This should be line 1.\n");

I certainly don't see many programs breaking because of this ...
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 385EAD11.17A3BE8C@cygnus.co.uk >you write:
  > Or a branch? That would keep it out of the reach of normal users using the
  > CVS tree, even with -fnew-abi, while still permitting collaboration - which
  > is presumably the reason Mark wants it checked in to the gcc repository.
Not a bad idea.  On a branch would be fine by me.  We have used that scheme
for similar purposes in the past.

jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@cygnus.com> writes:

    >> From: Mark Mitchell <mark@codesourcery.com> Date: Sun, 19 Dec
    >> 1999 22:26:18 -0800

    >> However, libio has to change to match the new ABI.  That's a
    >> fact.  Obviously, this will be a conditionally compiled change;
    >> the compiler's going to be supporting the old ABI indefinitely.

    Geoff> I think you're misunderstanding the point.  You can't

I don't think so.

    Geoff> conditionally _compile_ anything.  You must make sure
    Geoff> everything will work at runtime, out of the same objects.
    Geoff> This is what 'binary compatibility' means.

Yes.  Let's try this again.

Let's suppose we're running on IRIX.  There's no libio in libc.

  o Now, we can compile with the old C++ ABI or the new C++ ABI.  
    Either is OK -- there's no standard C++ ABI on IRIX.  The user
    might choose this which ABI to use at compiler-configuration time.

    But, the libio in libstdc+++ needs to know about the C++ ABI
    (that's a design choice in libio).  So, we need to make the libio
    in libstdc++ do its by-hand layout of C++ objects in the same 
    way as the C++ compiler.  Which way to do this is known to us
    when we build libio; we've already committed to the C++ ABI to
    use.  Depending on which way the user configured the compiler,
    we'll end up with different versions of libio.
    
  o On Linux, life is more difficult -- libstdc++ and glibc know
    about each other.

    I was thinking that *in principle* one could set up libio
    in libstdc++ analagously to the IRIX situation.  In particular,
    without knowing glibc, just as we do on IRIX.  Of course, one
    would only do this for the new ABI, and only when the system
    libc doesn't already contain the new libio.

    It may be that this is technically infeasible.  It's certainly
    not wrong in principle -- it's in fact the only way to let people
    use the new ABI without upgrading their C library.

To quote Nixon :-), "let me make one thing perfectly clear":

  o I'm not proposing changing *anything* about libio when compiled
    with the existing C++ ABI, which will continue to be supported
    indefinitely.

  o I'm proposing chaning libio, conditionally, when compiled with the
    new C++ ABI.  (This is analagous to multilibbing.)  Without doing
    so, it will not work.

The entirety of a single C++ program must be compiled using -fnew-abi,
or -fno-new-abi -- you can't compile bits and pieces one way.  If you
want to have a library that can be linked with the old ABI and the new
ABI, you compile it twice: once to support the old ABI and once to
support the new one.

That's no different for libio, since it is emulating the C++ ABI, even
though it is written in C.

    >> It would be nice if people can use the new ABI, if only to test
    >> it, on IA32 Linux.  They can't do that unless we can produce a
    >> version of libio that doesn't interact with the C library,
    >> since we can't expect everyone to go upgrading their C library.

    Geoff> Why not?  If the new ABI is stable (or stable enough for
    Geoff> libio), there is no reason not to simply tell people "gcc
    Geoff> version foo requires glibc version bar".

That's not very polite.  That means people with legacy Linux systems
can't use the advances in the compiler.  But, what you say is
possible.

However, we *still* have to change libio for other, non-Linux systems
-- or else nobody can use the new ABI *anywhere*.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jamie Lokier]

Per Bothner wrote:
> Jeffrey A Law <law@cygnus.com> writes:
> 
> > While I believe it was an unbelievably dumb thing for the glibc folks to
> > tie libio into the C library,
> 
> In other words, you don't think it is a reasonable goal to allow C
> stdio FILEs to be shared with C++ iostream streambufs?

The down side is it's not at all portable, so code that's written on
Linux will output differently on other platforms.

> The best way to synchronize (say) stdout with cout seems to be for
> them both to use the same buffer.  The requires (at least if you want
> efficiency) for them to use compatible data structures,

Only the /very/ few fields that are used by inline input/output
functions need to be shared.  Since they are simple data fields, sharing
those would be very easy even with many different kinds of C++ ABI.

Everything else, including vtable format and placement does not need to
be shared.  Just have the C library call functions (not virtual ones)
like "flush_my_output_stream" etc., and those functions would contain
the vtable lookup code.  Or alternatively, have an explicit /C/ vtable
as one of the shared fields and keep it that way for the C++ version.
C++ streams don't have to call that vtable; they merely have to provide
it.

-- Jamie

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jamie" == Jamie Lokier <jamie.lokier@cern.ch> writes:

    Jamie> <evil> You might be able to do it by reserving a field in
    Jamie> the new structures at the right place to contain the
    Jamie> old-Glibc-compatible vtable pointer, and having two

Interesting idea.  I think this might have been what Ulrich was
suggesting as well in one of his messages, but it didn't make sense to
me at the time.  Something like this might indeed help solve some of
the problems.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.

Wait a minute, why do you think you can change libio?  If something
has to be behave different there must be a different solution found.
You are not with the compiler, don't think you have the right to
change everything.  These things should better be coordinated with me.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Joe Buck]

I wrote:

> If glibc and libstdc++ are to share the same structures for streams/stdio,
> then yes, they must change in lock-step.  This is appropriate for
> *released* versions.

Jeff writes:

> And do you happen to know when the next release cycle is going to start?  Are
> you going to volunteer to remove this code if the release cycle starts before
> glibc & gcc have merged their libio implementations?  Are you going to
> volunteer to merge the two versions if Mark & CodeSourcery ultimately do not
> do the work?

Is this rhetorical or do you mean it?  In case you do mean it:
If that extremely unlikely event occurred, I'd just ship the thing and no
one would care.

The reason no one would care is this: users who do not specify -fnew-abi
would not see any difference, as Mark's changes would be #ifdef'd out.
(Mark's proposal results in NO CHANGE unless libstdc++ is built with
-fnew-abi!).

People using -fnew-abi would notice that some programs that mix stdio
and iostreams don't work, and that more memory is consumed due to two
libio variant copies.  This would be suboptimal but OK for a non-default,
unfinished feature.

Nevertheless, we're in violent agreement: the way to proceed is to start
out with a separate branch, then work on patches that all can accept.

Re: Linux vs. libio

[Linus Torvalds]

In article < 19991221155340.P14133@namaste.lokigames-lan.com >,
Michael Vance  <briareos@lokigames.com> wrote:
>
>What does bother me is when we agree that things won't break, and
>then they do. I expected things to break from libc5 to libc6. No
>pointing whining about it. I did not expect things to break from glibc
>2.0 to 2.1. Perhaps that was my misunderstanding.

Excellent point.

I personally disagree completely with the glibc approach of "backwards
comaptibility at all costs". 

It is _much_ preferable to be clearly new and different than to try to
be 100% compatible and succeeding to 99.9%. 

If there are major C++ internal ABI changes in gcc, glibc should just be
upgraded to the next major version, and all backwards compatibility
cruft should be removed.  People still have the old libraries installed,
and every single old binary works fine - simply because nothing has
changed as far as they know or care. 

Why are people discussing all these clever dual-vtable things at all?
Talk about subtle bugs..

I'm personally quite ashamed that people point to "Linux" as holding
things back (and I agree with them). Why is gcc so tightly linked to
glibc issues, and why can't glibc just change?

		Linus

Re: Linux vs. libio

[Jeffrey A Law]

  In message < u9g0wxbk5g.fsf@yorick.cygnus.com >you write:
  > >>>>> Jeffrey A Law <law@cygnus.com> writes:
  > 
  >  > You will need to work with the glibc folks to find a solution that makes
  >  > sense to them as we do not want to start changing the API of the libio
  >  > contained in GCC without making corresponding changes to the libio
  >  > contained in glibc.
  > 
  > The problem is that part of the libio API is the C++ vtables, so any ABI
  > change that modifies the layout of the vtable (such as -fvtable-thunks) or
  > how the vtable is found breaks libio. 
Right.  This is precisely why I stated that we should not change libio without
changing glibc too.  They need to change in lock-step.

That means that the libc folks have to be intimately involved in the decisions
we make regarding the ABI/API of libio (and IMHO have the ultimate yea/nay
decision for such changes).

  > This has caused us a lot of trouble over the years; perhaps the answer is
  > to change that design,
Possibly, but that does not help us with the existing glibc installations.

  > but this *is* supposed to be the all-singing, all-dancing, ultimate C++
  > ABI.
:-)  I'll be floating on air when it's all ready.  I really want to put
the ABI issues behind us once and for all.



jeff

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> Maybe I am missing something but couldn't the io code in linux be
> written in pure C

I believe it is.

> with a few extensions to allow C++ wrapper classes to
> effectively use the io services.   Like say provide inline functions (or
> macros) to manipulate the buffer efficiently.

It is more complicated than that.  The point is *extensibility*
through jump tables (in C) or vtables (in C++).  The standard
C++ libary is a set of classes (and templates) that use
inheritance and virtual functions.

For exmaple, the C++ version of `getc' has an inline path that
is fast and similar to C:
        if (str->read_pointer < str->read_limit)
          return *str->read_pointer++;
        else
          return handle_slow_case(str);

So what we in GNU is use the *same* inline code for both C and C++.
I.e. the C++ methods and the C getc access the *same* read_pointer.

For C++, handle_slow_case(str) actually does a virtual function
call to a method which depends on the run-time type of (*str)
(which is not necessarily known at compile-time).  For example,
we may be reading from a file, a string, a socket, a wrapper
around readline, or an Emacs-style buffer.  In standard C,
there is only only kind of FILE, so we can call the appropriate
function directly.  However, glibc has long provided similar
extensible I/O as C++, but using jump tables of function pointers.
It makes sense to use the same format for those jump tables as
the C++ virtual function tables, hence some of the ugly macros
in the libio source.

One can write a C++ wrapper class (say stdiobuf) that implements the
C++ functionality, but uses the C stdio buffers.  The problem is that
this is slow, since you have to call handle_slow_case on every
character.  You can force that by setting read_pointer == read_limit
(for those streams that are aliased to stdio).  Alternatively, you can
make sure to use the same value of read_pointer for C FILEs and C++
filebufs.  However, then either you get extra overhead (indirection)
or you have to merge the data structures for C and C++.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210029.QAA17146@atrus.synopsys.com >you write:
  > Mark Mitchell writes:
  > 
  > >   > If that's what the steering committee, or whoever decides this kind o
  > f
  > >   > thing, wants to do, then that's what we'll do.
  > 
  > Jeff writes:
  > > A branch is certainly the direction I'm leaning.
  > 
  > OK, how about this.
  > 
  > Step 1: Mark checks in the code as a branch.
  > 
  > Step 2: Mark works with Uli to find an patch that the glibc team can
  > accept, giving the *option* (not on by default) of separate
  > glibc/libstdc++ I/O structures.  The goal is to be able to run with
  > the production glibc and -fnew-abi for C++.
  > 
  > (glibc folks: please be open to those gcc developers and testers who want
  > to work with a stable glibc and an experimental libstdc++ on Linux.
  > Without it, we're never going to get the libstdc++ quality up.)
  > 
  > Step 3: Once agreement is obtained, the code is put into the main gcc branc
  > h.
Sounds reasonable to me.

  > >   >   o We're not going to do stuff willy-nilly without asking the libio
  > >   >     folks for approval.
  > > This is the issue.  And I've stated before that if you get a buy-in from
  > > the glibc folks then you can go forward.  But you need a buy in *before*
  > > you start checking in the changes.
  > 
  > Fine (if "checking in" means "checking in on the main branch").
Yes.  That is precisely what I meant.  I should have been clearer on that 
point.

jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@cygnus.com> writes:

Thanks for your email.  You were very clear.

    Geoff> So we have a shared library use graph that looks like this:

    Geoff> netscape --> libstdc++.so.2.8 --> libc.so.6 test -->
    Geoff> libstdc++.so.3 ----^

    Geoff> But this won't work with your proposal.  I can have the
    Geoff> libio in libc.so.6 compiled with the new C++ ABI, in which
    Geoff> case the old libstdc++ won't work, or it can be compiled
    Geoff> with the old ABI in which case the new libstdc++ won't
    Geoff> work.

Right; that's why I was proposing figuring out an *optional* way of
divorcing libio from libstdc++ -- exactly to avoid this problem.  I
understand that's not your long-term goal, but I wanted to support
exactly this situation.  So, I envisioned all of:

  o libstdc++ with old ABI working with old libc.
  o libstdc++ with new ABI working with new libc.
  o libstdc++ with new ABI working with old libc.
  o libstdc++ with old ABI working with new libc.

    Geoff> So we need a libio in glibc which will work for _both_ the
    Geoff> old ABI and the new ABI.

If we can do this, that will be wonderful.  I'm in no way opposed to
this, if it is technically feasible.  I'll try to see how this might
be done.

In case you missed the earlier discussion, one of the issues is that
things like _IO_XSPUTN don't work because the vtable pointer is now
before the _IO_FILE, not after it.

As was suggested, for example, the vtable pointer issue can probably
be solved by having *two* vtable pointers in the new objects -- the
one the C++ ABI requires, and the one the libio routines expect.  I
think that's the *only* way to solve that particular problem, and I
expect there will be other problems as well.

But, one of the vtables will contain thunks that adjust the this
pointer before calling through the real vtable.  So there will be two
costs to just this level of compatibility: extra space, and extra
time.  That seems undesirable to me.

Can you suggest a way of making this work that doesn't involve a
performance hit?

    Geoff> more complicated if you only want one library.  Hopefully
    Geoff> the new C++ symbol mangling and the old one don't overlap
    Geoff> so the 'more complicated' just means specifying both
    Geoff> versions of the object files on the link line.

The mangling has not yet been decided, so that's unclear.  But note
that, in your proposal, even the C parts of libio need to handle both
ABIs, so I don't think mangling is enough to keep things totally
separate.

    Geoff> libio is special because it is part of glibc.  The C++ ABI
    Geoff> changes at pretty frequent intervals, the C ABI doesn't.

It is the intention of myself (and I believe Jason Merrill would
agree, after the new ABI is implemented, never to change it again --
just as you folks have committed to with libc.  We recognize that one
of the obstacles to widespread C++ acceptance has been the lack of a
stable ABI, and I'm hoping that the new ABI, which has been developed
by a committee of intelligent people, will be good enough that we can
stick with it indefinitely.  Time will tell, of course.

One of the things that has happenned up until now is that new features
of C++ have required ABI changes.  The new ABI does explicitly deal
with all the lanague features.  My primary fear is not inadequacies in
the ABI -- it is bugs in the ABI implementation in the compiler.

    Geoff> This is not a new thing.  People with legacy Linux systems
    Geoff> can't use advances in the kernel unless they upgrade their
    Geoff> libc, nor can they run programs that were built under newer
    Geoff> versions of the libc.  In this case, they just won't be
    Geoff> able to use the new ABI until they upgrade, which probably
    Geoff> won't cause them much pain.

One of my concerns is testing: I want people to be able to test the
new ABI.  Lots of those potential testers are on IA32 Linux, but they
don't really want to upgrade their systems, or even have to install
new versions of libc in funny hidden places just to do the testing.

But I think you and I will have to agree to disagree on this point; I
think we just evaluate the likelihood of people being willing to
upgrade differently.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Geoff Keating]

> Date: Tue, 21 Dec 1999 14:24:19 -0500 (EST)
> From: Kevin Atkinson <kevinatk@home.com>

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

On my machine, I/O from gas and gcc is compute bound.  I expect it's
the same on most systems.  Modern caching techniques are pretty good,
and there aren't really many programs which have a working set of more
than 100Mb while compiling (and then it's usually because GCC is being
a glutton, not because of the size of the sources and objects).

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> If you want I'll ask for a formal vote from the steering
    Jeffrey> committee.

No, that will not be necessary.
 
We've discussed this internally at some length.  Since we're not
obligated to do the libio work at all (that was merely volunteerism on
our part, for the sake of the community at large), we've elected not
to do it.  

We'll just stick to our job: implement the new ABI support.  We'll do
that on a branch.  I belive it would be inappropriate to do the
-fnew-abi work on the mainline since that will be an incomplete
change: changing -fnew-abi and not changing libio will result in a
libstdc++ that does not work with -fnew-abi.  Of course, we will make
reasonable efforst to ensure that generic changes that come about as
part of this work appear in the mainline as well as on the branch.

It is our hope that some entity (perhaps CodeSourcery, if a customer
requests it) will eventually do the libio work, as well as the work
required to merge the -fnew-abi changes into the mainline.

I thank all of you for your time, ideas, comments and criticisms, and
I hope that the archive of this debate will assist the next person who
tackles this problem in formulating an elegant solution to the various
libio issues.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Wait a minute, why do you think you can change libio?  If
    Ulrich> something has to be behave different there must be a
    Ulrich> different solution found.  You are not with the compiler,

We need a procedural clarification here.

I assumed that "blanket write privs" applied to everything you get
when you do a `cvs co egcs', which includes libio and libstdc++.

Obviously, making changes there, as with any part of the system,
requires care, and entails risk.  Just as I would not go willy-nilly
hacking around in the ARM back-end (which I know next to nothing
about), I would not go willy-nilly hacking about thinks in libio.
And, were I to change the ARM back-end, and were Richard Earnshaw to
tell me I had done so incorrectly, I would certainly defer to his
judgement.

But, there's still a question remaining: are people with blanket write
privs allowed to change libio/libstdc++ without your approval, or not?
And, for that matter, it appears that Gabriel Dos Reis is also listed
as a maintainer of "c++ runtime libs" as well, so may I assume that
even if approval is required, Gaby's approval is good enough?

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

>     Joe> I think that the glibc folks should
>     Joe> approve patches in the libraries with ABI-visible effects
>     Joe> before they go to release branches, but we shouldn't need to
>     Joe> wait for such approval to check something into the CVS
>     Joe> archive.  After all, some problems won't be detected until
>     Joe> people test snapshots anyway.

Mark Mitchell writes:

> Thanks for clarifying!

Please note that I'm only one SC member.  If other members object to
what I'm saying above, we'd have to resolve it.

Re: Linux vs. libio

[Joe Buck]

>   > It isn't really required that libio and glibc change in lockstep.  It
>   > certainly doesn't matter on platforms that don't use glibc.

> Huh?  If you don't change them in lockstep you introduce incompatibilities
> between the version in the linux C library and the version in GCC.  That
> has proven to be a majorly stupid thing to do.  We don't want to repeat
> that mistake again.  So I'll repeat.  They must change in lock step.

If glibc and libstdc++ are to share the same structures for streams/stdio,
then yes, they must change in lock-step.  This is appropriate for
*released* versions.  But for *testing*, it would be nice to have the
*option* on Linux of saying "don't share i/o structures".  Enabling this
option would break programs that rely on the C++ standard's guarantee that
cout/stdout and cerr/stderr are synchronized (unless flush calls are
inserted at appropriate points), and we would never build binary packages
to be distributed in that way, though we could point users to this choice
if they are stuck, trying to get new C++ code to run on a legacy Linux
system, for instance.

Ideally, this option would not represent a fork because we could find
an agreed-apon way to do this (agreeable to both the C++ and glibc teams).

>   > Why should we make it hard for people to test the new ABI?
> Why should we diverge from glibc?  That just makes our life more difficult.
> 
> You've got find another way which doesn't cause our libio to diverge from
> the one in glibc.  No ifs ands or buts about it.

It appears that the only way to get past this impasse is to find a libio
patch that the glibc folks would be willing to accept.

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991221155340.P14133@namaste.lokigames-lan.com >you write:
  > > Somehow, I don't quite believe that, no matter how well-meaning the glibc
  > > folks have been, such disasters won't happen again...  
  > 
  > Neither do I. Sure, I'd love for the C++ ABI to settle, but I'm not
  > counting on it.
At some point it really does need to settle.  IMHO, the biggest reason it
hasn't settled is the major vendors have all been targetting their own
cpus (sparc, parisc, mips, powerpc, etc) and as a result there hasn't
been much interest in ABI work to allow code built with one vendor's tools
to interoperate with code built by another vendor's tools.

That's starting to change with ia64.  Suddenly a number of players want to
have some level of binary compatibility with each others tools, so of
course they have start defining an ABI.

It's been a long time coming, but I'm happy to see progress made.

  > > linux folks (like having to upgrade the glibc version number...), will th
  > > Steering Committee take steps to ensure the Right Thing is done ?  
  > 
  > I'm concerned about this, too.
Likewise.

  > I develop binary only software for Linux. It is a pain
  > (__register_frame_info, glibc 2.0 vs 2.1 ABI breakage, etc.).
Yes.  These have all been a major pain in the (*&@#$.  I'm not at all
happy with how these issues have been handled to date.

  > First and foremost, I want a compiler that works. If it's possible to
  > do that within the framework of backwards compatability, so much the
  > better--as long as it isn't a burden on further development.
Certainly we all want a compiler that works.  No doubt.  But if we're
going to be taken seriously we also have to spend a lot more time
dealing with backwards compatibility than we have in the past.   In fact,
IMHO, it was a lack of attention to backwards compatibility and
communication between gcc & glibc that lead to the ongoing 
__register_frame_info problems!

  > then they do. I expected things to break from libc5 to libc6. No
  > pointing whining about it. I did not expect things to break from glibc
  > 2.0 to 2.1. Perhaps that was my misunderstanding.
This is largely out of the control of the GCC project since we do not
control glibc.


jeff

Re: Linux vs. libio

[Geoff Keating]

> From: Joe Buck <jbuck@synopsys.com>
> Date: Mon, 20 Dec 99 14:42:50 PST
> Cc: jbuck@synopsys.com, mark@codesourcery.com, drepper@cygnus.com,
>         gcc@gcc.gnu.org, libc-alpha@sourceware.cygnus.com
> 
> > My opinion:
> > 
> > The primary purpose of snapshots are for use in the testing and
> > further development of glibc.
> 
> Sorry for the lack of clarity (given that this is going to two lists):
> I was talking about gcc/egcs snapshots, which need to be able to test
> the new C++ API.  We aren't talking about glibc snapshots here.

Did I write that?  I meant, "...development of GCC", I was thinking
about GCC snapshots.  Of course, the rest of my e-mail seems to apply
equally well to both.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@envy.cygnus.com> writes:

    Geoff> It may be that you will have to change glibc to be
    Geoff> compatible with the new layout.  You've already changed
    Geoff> glibc to be compatible with the new EH so you will be
    Geoff> requiring a new version of glibc anyway.

I think you mean the new atexit stuff.  Technically, that can go in
libstdc++, if necessary, and just replace the bits in libc.

But, you are correct -- in the end there will need to be a new libc.
It may or may not be deployed on IA32 Linux; that's not something I'll
decide.

However, I'm sure it will be deployed on IA64 Linux.  It's also likely
to be used on existing non-Linux platforms, like Solaris or IRIX.

Right now, I'm just trying to test the work, and the systems I have
handy are IA32 Linux.

    Geoff> You don't want to do that.  It would cause C stdio and C++
    Geoff> iostreams to be buffered separately causing output to be
    Geoff> interleaved wrongly.  It would also break the nice
    Geoff> stdio/iostreams interface that took so much work to get
    Geoff> right.

All these things are no doubt true.

I know why there's interdependency between these things, and I know we
want to keep that in the future.

However, libio has to change to match the new ABI.  That's a fact.
Obviously, this will be a conditionally compiled change; the
compiler's going to be supporting the old ABI indefinitely.  

It would be nice if people can use the new ABI, if only to test it, on
IA32 Linux.  They can't do that unless we can produce a version of
libio that doesn't interact with the C library, since we can't expect
everyone to go upgrading their C library.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Michael Vance]

On Wed, Dec 22, 1999 at 12:03:21AM +0100, Marc Espie wrote:

> I've followed the glibc issues from the sidelines.

As have I. I should note, for fairness sake, that I am a customer of
Mark's.

> Somehow, I don't quite believe that, no matter how well-meaning the glibc
> folks have been, such disasters won't happen again...  

Neither do I. Sure, I'd love for the C++ ABI to settle, but I'm not
counting on it.

> linux folks (like having to upgrade the glibc version number...), will the
> Steering Committee take steps to ensure the Right Thing is done ?  

I'm concerned about this, too.

I develop binary only software for Linux. It is a pain
(__register_frame_info, glibc 2.0 vs 2.1 ABI breakage, etc.). But, it is
not nearly as much a pain as having to track down very weird bugs in
the compiler (the atexit() crap for static object destructors in
shared libraries). Or having to find/pay someone who is
qualified/interested/etc. to do the work on the compiler. First and
foremost, I want a compiler that works. If it's possible to do that
within the framework of backwards compatability, so much the
better--as long as it isn't a burden on further development.

What does bother me is when we agree that things won't break, and
then they do. I expected things to break from libc5 to libc6. No
pointing whining about it. I did not expect things to break from glibc
2.0 to 2.1. Perhaps that was my misunderstanding.

> * companies that distribute binary-only applications.

As above, it doesn't help us if the compiler is broken.

m.

-- 
Programmer                         "I wrote a song about dental floss,
Loki Entertainment Software         but did anyone's teeth get cleaner?"
http://lokigames.com/~briareos/              - Frank Zappa, re: the PMRC

Re: Linux vs. libio

[Marc Espie]

In article < Pine.LNX.4.10.9912211422460.7718-100000@kevins-linux >,
Kevin Atkinson <kevinatk@home.com> wrote:
>On 21 Dec 1999, Per Bothner wrote:

>> Kevin Atkinson <kevinatk@home.com> writes:

>> > >From http://www.sgi.com/Technology/STL/standard_library.html:

>> > Which is what I was trying to get at.  By directly maniplulating FILE
>> > one avoids the extra overhead.

>> Wise man say: "Every problem in computer science can be solved
>> by adding an extra level of indirection."   While SGI's technique
>> is clever, it is basically just another layer of indirection.
>> Specifically, you get at least two extra memory de-references
>> even in the fast case.

>Will that extra level of indirection *really* make a diffrence in speed I
>mean IO is something which is fairly slow in the first place.

It does make a *real* difference.  
I have some hard measurement data where I shaved 20% off a program's
execution time by getting data line by line and managing the buffer myself
instead of using fgetc.

Linux vs. libio

[Mark Mitchell]

Folks --

  Does anyone know how, on Linux, to build the libio (in CVS with GCC)
in such a way as to be entirely disentangled from the C library?
(Except in the ways that libio would normally be entangled with the C
library on any non-glibc system.)

  In particular, we're working on the new C++ ABI.  That requires some
changes to libio to deal with different class layouts, etc.  That's
mostly working, but the stdio functions in glibc have a bad tendency
to call functions in the modified libio as callbacks, and those
functions now behave differently than glibc expects.

  In other words, what I'm looking to do is to have two distinct,
disentangled, copies of libio: one in libc, to handle stdio, and one
in libstdc++ to handle iostreams.  How can this be done?

  Thanks,

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> But, there's still a question remaining: are people with blanket write
> privs allowed to change libio/libstdc++ without your approval, or not?

I certainly reserve all rights for the libio parts as I'm probably the
only one who sees the whole picture (libio in libstdc++ and in glibc).
As for the rest of libstdc++, it's not me.  Rather Benjamin and
Gabriel, so they have to speak for themselves.  And yes, I think the
write access you got from Jeff and the gcc people does not apply to
the libraries.  Esp libio has far too many implications then you can
test for by simply running libstdc++ tests (not that there were enough
tests in the first place).  Plus: the version in libstdc++ is not the
latest one (i.e., not the development version).

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    >> The problem is that part of the libio API is the C++ vtables,
    >> so any ABI change that modifies the layout of the vtable (such
    >> as -fvtable-thunks) or how the vtable is found breaks libio.

    Jeffrey> Right.  This is precisely why I stated that we should not
    Jeffrey> change libio without changing glibc too.  They need to
    Jeffrey> change in lock-step.

We're not talking about a release here.  We're just talking bout
development.

It isn't really required that libio and glibc change in lockstep.  It
certainly doesn't matter on platforms that don't use glibc.

What's required is that if libio changes on IA32 Linux (which, under
my proposal, it would only do when compiled with -fnew-abi), then it
needs to be simultaneously disentangled from glibc, and work in the
way that libio works with other C libraries.  If that disentanglement
can't be done, then it can't be done -- people will only be able to
use -fnew-abi on non IA32 Linux platforms.

If we decide to *release* an IA32 Linux compiler where -fnew-abi is a
supported option (which we have not yet done to date), then we
probably want to do so only when there's a C library that has the new
libio.  That's not a requirement for releasing an IRIX compiler where
-fnew-abi is a supported option.

It's very hard to move in lock-step; I think there are IA32 Linux
users who would like to try out -fnew-abi (to see if it works, to see
if it performs better, etc.), but who don't want to upgrade their C
libraries.  (I'm one of those users.)  

Why should we make it hard for people to test the new ABI?

There are really only a few options that I can think of:

  1. Don't implement -fnew-abi.

  2. Implement -fnew-abi, but ignore the fact that libio is broken
     for the new ABI.  

     This solution makes -fnew-abi difficult to test, and makes
     an -fnew-abi compiler useless to the community, unless they 
     get iostreams from somewhere else.  (For example, SGI has
     a conforming implementation available at
     http://www.sgi.com/Technology/STL .)

  3. Implement the libio changes in the glibc version.  That will 
     make it considerably more difficult to test the work we are
     really trying to do: implement the new ABI.  Just mucking with
     the build process in GCC to make this work will be a
     nuisance.  Then, we have to either wait until the libio in glibc
     is considered stable and move the result back into the compiler,
     or backport the changes to the version in the compiler.

  4. Implement the libio changes in the compiler version.  That
     makes testing easy, and doesn't risk changing the libio to 
     a newer, less stable version.  Then, forward-port the changes
     from there into glibc.

Option 4 makes sense.  It requires minimal changes.  It allows us to
implement and experiment with -fnew-abi most effectively.  Once things
are working smoothly, we can move the changes into glibc.  We'll
commit to helping with that process.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Per Bothner <per@bothner.com> writes:

> So whether "vtable compatibility" is an issue or not depends
> on which extended glibc-specific features are a documented
> part of the API and how programs access these features.

The problem is not the glibc code itself.  You are right, we can
change the vtables and it should not matter since none of these things
are documented.

The problem is passing C++ iostream classes to the functions in genops
and fileops.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

> I assumed that "blanket write privs" applied to everything you get
> when you do a `cvs co egcs', which includes libio and libstdc++.

My opinion:

Blanket write privs are for the purpose of checking things in to CVS.
Actually doing a release is another matter: for anything that affects
other projects, the Steering Committee would have to be involved,
and the SC tries to run by consensus, meaning that if Ulrich Drepper
were extremely unhappy about something, the SC wouldn't approve its
release.

But remember, we never promised that we wouldn't break things in
*snapshots*.  That's too much of a constraint.

> But, there's still a question remaining: are people with blanket write
> privs allowed to change libio/libstdc++ without your approval, or not?
> And, for that matter, it appears that Gabriel Dos Reis is also listed
> as a maintainer of "c++ runtime libs" as well, so may I assume that
> even if approval is required, Gaby's approval is good enough?

I think we need a compromise between measures that would slow down
development too much, and measures that would break things for users.  I
think that the glibc folks should approve patches in the libraries with
ABI-visible effects before they go to release branches, but we shouldn't
need to wait for such approval to check something into the CVS archive.
After all, some problems won't be detected until people test snapshots
anyway.

Re: Linux vs. libio

[Ulrich Drepper]

Jeffrey A Law <law@cygnus.com> writes:

> I don't really see the need to debate this -- it's a fact of life that we
> have to deal with this braindamage.

You don't want any discussion but still insist on insulting other
people.  Very strange.  Plus that you keep saying this does mean you
haven't spend a second thinking about it.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> Fortunately, the changes are relatively minimal: almost no code --
> just a few changes in structure definitions and in a few macros.

This is not the point.  As Geoff pointed out the situation on Linux at
least is therefore wrt C++ I/O pretty good because of the tight
integration of the C and C++ functions.  Besides this is much more
light-weight and in future (after the merge with my development
version) it'll allow full wide-char and locale integration, something
which cannot 100% be achieved on other platforms.  This is not
possible if you now simply change the API and actually generate libio
code in libstdc++ which does not use the code in the libc.

Adding these changes will probably be ok for other platforms (since
there is no backward compatibility to be maintained) and it is also ok
*for test purposes* to use this changed code on Linux, but no
production release should be made with these changes and people should
be discouraged to distribute libstdc++ versions compiled in this way.
libstc++ is for various reasons generated most often as a shared
object and therefore it is really A Bad Thing(tm) to change the API in
an incompatible way just to change it again a bit later.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Kevin Atkinson]

On 21 Dec 1999, Per Bothner wrote:

> Kevin Atkinson <kevinatk@home.com> writes:
> 
> > >From http://www.sgi.com/Technology/STL/standard_library.html:
> > ...
> > Which is what I was trying to get at.  By directly maniplulating FILE
> > one avoids the extra overhead.
> 
> Wise man say: "Every problem in computer science can be solved
> by adding an extra level of indirection."   While SGI's technique
> is clever, it is basically just another layer of indirection.
> Specifically, you get at least two extra memory de-references
> even in the fast case.

Will that extra level of indirection *really* make a diffrence in speed I
mean IO is something which is fairly slow in the first place.

--
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Per Bothner]

Mark Mitchell <mark@codesourcery.com> writes:

> As was suggested, for example, the vtable pointer issue can probably
> be solved by having *two* vtable pointers in the new objects -- the
> one the C++ ABI requires, and the one the libio routines expect.  I
> think that's the *only* way to solve that particular problem, and I
> expect there will be other problems as well.

Well, for C code, it is not even clear you need to maintain
compatibility in terms of vtables.  As far as I know, of the
public C header files (such as stdio.h or libio.h) only call
named function;  calling a "method" (a slot in a jump table
or vtable) is only done internally.

So whether "vtable compatibility" is an issue or not depends
on which extended glibc-specific features are a documented
part of the API and how programs access these features.
There is also a slight danger of programs that use undocumented
backdoor features and functions;  that is what testing (such as
re-building all of say Debian) is for.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Joern Rennecke]

> Jeffrey A Law <law@cygnus.com> writes:
> 
> > I don't really see the need to debate this -- it's a fact of life that we
> > have to deal with this braindamage.

Ulrich Drepper:
> You don't want any discussion but still insist on insulting other
> people.  Very strange.  Plus that you keep saying this does mean you
> haven't spend a second thinking about it.

C++ was known to be volatile when libio was written.
The compatibility issues we have now could have been avoided if the API
of libio had been designed to use plain C.

Well, now that the damage is done, how about introducing an attribute
for functions and structures that the old ABI is to be used.
The header files that declare libio functions and structures for other
code to use could bear this attribites, so that we can freeze the libio
API while the default ABI changes.

Re: Linux vs. libio

[Joe Buck]

> My opinion:
> 
> The primary purpose of snapshots are for use in the testing and
> further development of glibc.

Sorry for the lack of clarity (given that this is going to two lists):
I was talking about gcc/egcs snapshots, which need to be able to test
the new C++ API.  We aren't talking about glibc snapshots here.

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Sun, 19 Dec 1999 22:26:18 -0800

> However, libio has to change to match the new ABI.  That's a fact.
> Obviously, this will be a conditionally compiled change; the
> compiler's going to be supporting the old ABI indefinitely.  

I think you're misunderstanding the point.  You can't conditionally
_compile_ anything.  You must make sure everything will work at
runtime, out of the same objects.  This is what 'binary compatibility'
means.

> It would be nice if people can use the new ABI, if only to test it, on
> IA32 Linux.  They can't do that unless we can produce a version of
> libio that doesn't interact with the C library, since we can't expect
> everyone to go upgrading their C library.

Why not?  If the new ABI is stable (or stable enough for libio), there
is no reason not to simply tell people "gcc version foo requires glibc
version bar".

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich\" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich\> Wait a minute, why do you think you can change libio?  If

Because it's under the GPL? :-) :-) :-)

Please don't get territorial.  I'm not about to go releasing some
weird forked version of anything.  I asked you for help because I know
you're the expert.

I'm just doing work.  You can always choose to accept or reject any
changes in the end; that's your privilege as libio maintainer.

    Ulrich\> something has to be behave different there must be a
    Ulrich\> different solution found.  You are not with the compiler,

There *is* going to be a new C++ ABI.  We're all agreed on that, and
we're agreed on what it will be.  It's not some whim of mine; it's an
industry-standard cross-OS ABI.

The libio tricks whereby there are C versions of C++ objects (complete
with vtables, etc.) will still work, but the objects are going to have
to be laid out differently, because the C++ objects are being laid out
differently.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[H.J. Lu]

> 
> 
> Folks --
> 
>   Does anyone know how, on Linux, to build the libio (in CVS with GCC)
> in such a way as to be entirely disentangled from the C library?
> (Except in the ways that libio would normally be entangled with the C
> library on any non-glibc system.)
> 
>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.
> 
>   In other words, what I'm looking to do is to have two distinct,
> disentangled, copies of libio: one in libc, to handle stdio, and one
> in libstdc++ to handle iostreams.  How can this be done?
> 

Take a look at how libc5 is handled in libio in gcc. libio in gcc
2.95.2 is quite different from libio in libc5.

-- 
H.J. Lu (hjl@gnu.org)

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

>   > If that's what the steering committee, or whoever decides this kind of
>   > thing, wants to do, then that's what we'll do.

Jeff writes:
> A branch is certainly the direction I'm leaning.

OK, how about this.

Step 1: Mark checks in the code as a branch.

Step 2: Mark works with Uli to find an patch that the glibc team can
accept, giving the *option* (not on by default) of separate
glibc/libstdc++ I/O structures.  The goal is to be able to run with
the production glibc and -fnew-abi for C++.

(glibc folks: please be open to those gcc developers and testers who want
to work with a stable glibc and an experimental libstdc++ on Linux.
Without it, we're never going to get the libstdc++ quality up.)

Step 3: Once agreement is obtained, the code is put into the main gcc branch.

>   >   o We're not going to do stuff willy-nilly without asking the libio
>   >     folks for approval.
> This is the issue.  And I've stated before that if you get a buy-in from
> the glibc folks then you can go forward.  But you need a buy in *before*
> you start checking in the changes.

Fine (if "checking in" means "checking in on the main branch").

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> This is still no reason to break compatibility.  If the
    Ulrich> virtual tables require a new layout one can add new ones.

OK, I'm listening.  I don't understand what you mean, but if you can
show me how to do this, and avoid having to have two separate libios
(one in libstdc++, and one in libc), I'll be all too happy to
implement it.

With the new ABI, pointers to virtual tables come at a different point
in the objects.  Bases are laid out in different orders.  There are no
longer pointers to virtual bases.  Some of that doesn't matter; some
of it does.  Given:

  struct _ios_fields { ... };
  class ios : public _ios_fields { ... };

it is no longer the case that you can cast an `ios *' to an
`_ios_fields *' without having to do modify the pointer value; the
vtable pointer comes before the _ios_fields object in an ios.

More importantly, the same holds for streambuf and _IO_FILE.  You
can't cast an _IO_FILE to a streambuf, or vice versa, without
modifying the pointer value.

That means the (old) macros (like JUMP1) that take an _IO_FILE *, go
find the vtable, and jump through it, don't work when given an
_IO_FILE * that came from a (new) streambuf; the vtable is now
*before* the _IO_FILE, not after.  Since there is code that uses those
macros already compiled into libc, I don't see how to make them
interoperate.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> That's the chance you and everyone must take when they
    Jeffrey> decide to work with free software projects.  If you don't
    Jeffrey> have the resources to do the merge, then you (in reality)
    Jeffrey> under-bid the work by under-estimating the integration
    Jeffrey> aspects of the contract. 

I don't think discussing these kinds of details on this list is
appropriate.  Our business practices are our business, just as Cygnus'
are yours.  They've no place here.

Integrating our changes is something we do voluntarily, as a service
to the community -- not because we have to.

I believe the proposal I made in my last email is reasonable, and I
believe it is the one whereby the community comes out the best: we
deliver a working libio and working new ABI, integrated on the
development branch of GCC, and the only cost is some temporary
instability in GCC when using -fnew-abi -- an already unsupported
option.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220150459I.mitchell@codesourcery.com >you write:
  > Let's suppose we're running on IRIX.  There's no libio in libc.
  > 
  >   o Now, we can compile with the old C++ ABI or the new C++ ABI.  
  >     Either is OK -- there's no standard C++ ABI on IRIX.  The user
  >     might choose this which ABI to use at compiler-configuration time.
  > 
  >     But, the libio in libstdc+++ needs to know about the C++ ABI
  >     (that's a design choice in libio).  So, we need to make the libio
  >     in libstdc++ do its by-hand layout of C++ objects in the same 
  >     way as the C++ compiler.  Which way to do this is known to us
  >     when we build libio; we've already committed to the C++ ABI to
  >     use.  Depending on which way the user configured the compiler,
  >     we'll end up with different versions of libio.
  >     
  >   o On Linux, life is more difficult -- libstdc++ and glibc know
  >     about each other.
  > 
  >     I was thinking that *in principle* one could set up libio
  >     in libstdc++ analagously to the IRIX situation.  In particular,
  >     without knowing glibc, just as we do on IRIX.  Of course, one
  >     would only do this for the new ABI, and only when the system
  >     libc doesn't already contain the new libio.
  > 
  >     It may be that this is technically infeasible.  It's certainly
  >     not wrong in principle -- it's in fact the only way to let people
  >     use the new ABI without upgrading their C library.
  > 
  > To quote Nixon :-), "let me make one thing perfectly clear":
  > 
  >   o I'm not proposing changing *anything* about libio when compiled
  >     with the existing C++ ABI, which will continue to be supported
  >     indefinitely.
  > 
  >   o I'm proposing chaning libio, conditionally, when compiled with the
  >     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
  >     so, it will not work.
  > 
  > The entirety of a single C++ program must be compiled using -fnew-abi,
  > or -fno-new-abi -- you can't compile bits and pieces one way.  If you
  > want to have a library that can be linked with the old ABI and the new
  > ABI, you compile it twice: once to support the old ABI and once to
  > support the new one.
  > 
  > That's no different for libio, since it is emulating the C++ ABI, even
  > though it is written in C.
  > 
  >     >> It would be nice if people can use the new ABI, if only to test
  >     >> it, on IA32 Linux.  They can't do that unless we can produce a
  >     >> version of libio that doesn't interact with the C library,
  >     >> since we can't expect everyone to go upgrading their C library.
  > 
  >     Geoff> Why not?  If the new ABI is stable (or stable enough for
  >     Geoff> libio), there is no reason not to simply tell people "gcc
  >     Geoff> version foo requires glibc version bar".
  > 
  > That's not very polite.  That means people with legacy Linux systems
  > can't use the advances in the compiler.  But, what you say is
  > possible.
  > 
  > However, we *still* have to change libio for other, non-Linux systems
  > -- or else nobody can use the new ABI *anywhere*.
All this is fine and good *as long as you can get Uli and the glibc folks to
agree to the change*.  I'm not going to support diverging our libio sources
from the glibc libio sources.  Even in the development tree.  That ultimately
is a lose.

I'll repeat, if you can get Uli & friends to agree to the change, then it can
go into the development sources.  If not, then it must not go into the
development sources (it can of course still go on a branch).


jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> If you want to update the version of libio in the compiler, by
    >> all means go ahead!

    Ulrich> I would have done this if it would make sense.  But it
    Ulrich> does not since the changes are not yet fully tested and
    Ulrich> probably not even complete.  Adding all the wide char
    Ulrich> stuff is non-trivial.

That's what I figured.  So, we're between the proverbial devil and the
deep blue sea: we can't easily test -fnew-abi without making some
changes in the version used with the compiler, and we can't easily
upgrade that version.  So, we'll have to make the changes, and merge
later, I think.

Fortunately, the changes are relatively minimal: almost no code --
just a few changes in structure definitions and in a few macros.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Mark" == Mark Mitchell <mark@codesourcery.com> writes:

    Mark> Fortunately, the changes are relatively minimal: almost no
    Mark> code -- just a few changes in structure definitions and in a
    Mark> few macros.

Actually, I lied. :-)  That's the conceptual picture -- there are
actually relatively many places where the code will get changed -- but
in a reasonably uniform way.

For example, we currently use the same set of macros (like _IO_XSPUTN)
on both `streambuf *'s and `_IO_FILE *'s.  That doesn't work any more
since the pointer types are not interchangeable.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

>   In message < 385EAD11.17A3BE8C@cygnus.co.uk >you write:
>   > Or a branch? That would keep it out of the reach of normal users using the
>   > CVS tree, even with -fnew-abi, while still permitting collaboration - which
>   > is presumably the reason Mark wants it checked in to the gcc repository.
> Not a bad idea.  On a branch would be fine by me.  We have used that scheme
> for similar purposes in the past.

Yes, this is the right solution.  People telling Mark to do the work in
his own tree are missing the point that no one can help him test.  But
we want to avoid disrupting others' work.

Re: Linux vs. libio

[Jeffrey A Law]

  In message < m21z8h1rmy.fsf@magnus.bothner.com >you write:
  > Jeffrey A Law <law@cygnus.com> writes:
  > 
  > > While I believe it was an unbelievably dumb thing for the glibc folks to
  > > tie libio into the C library,
  > 
  > In other words, you don't think it is a reasonable goal to allow C
  > stdio FILEs to be shared with C++ iostream streambufs? 
  > The best way
  > to synchronize (say) stdout with cout seems to be for them both to use
  > the same buffer.  The requires (at least if you want efficiency) for
  > them to use compatible data structures, and thus libio has to be tied
  > to the C library.  (Note: This is not new to glibc; it has been that
  > way in Linux for many years.)
I don't really see the need to debate this -- it's a fact of life that we
have to deal with this braindamage.  So, let's find a way to deal with it
rather than debating the decision to integrate libio with glibc here.

jeff

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 15:04:59 -0800

>     Geoff> conditionally _compile_ anything.  You must make sure
>     Geoff> everything will work at runtime, out of the same objects.
>     Geoff> This is what 'binary compatibility' means.
> 
> Yes.  Let's try this again.
...
>   o On Linux, life is more difficult -- libstdc++ and glibc know
>     about each other.
> 
>     I was thinking that *in principle* one could set up libio
>     in libstdc++ analagously to the IRIX situation.  In particular,
>     without knowing glibc, just as we do on IRIX.  Of course, one
>     would only do this for the new ABI, and only when the system
>     libc doesn't already contain the new libio.
> 
>     It may be that this is technically infeasible.  It's certainly
>     not wrong in principle -- it's in fact the only way to let people
>     use the new ABI without upgrading their C library.

In principle, it would be better to have libio and IRIX libc know
about each other.  Unfortunately, this isn't practical.  Under GNU
operating systems (OSs using glibc) this is possible, and very
desirable.

> To quote Nixon :-), "let me make one thing perfectly clear":
> 
>   o I'm not proposing changing *anything* about libio when compiled
>     with the existing C++ ABI, which will continue to be supported
>     indefinitely.
> 
>   o I'm proposing chaning libio, conditionally, when compiled with the
>     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
>     so, it will not work.

This is what I'm objecting to.  Perhaps I should draw a diagram.  I
expect that with -fnew-abi, the version number of libstdc++ will
change, perhaps to libstdc++.so.3.  Now, suppose I use my brand new
GCC 3.0.0, with -fnew-abi on by default, under my linux system,
to compile an executable 'test', and 'netscape' is an old binary-only
application compiled under the old GCC.

So we have a shared library use graph that looks like this:

netscape --> libstdc++.so.2.8 --> libc.so.6
test	 --> libstdc++.so.3   ----^

But this won't work with your proposal.  I can have the libio in
libc.so.6 compiled with the new C++ ABI, in which case the old
libstdc++ won't work, or it can be compiled with the old ABI in
which case the new libstdc++ won't work.

It is the glibc maintainers' policy to never again change the libc
version number if we can possibly help it.  Bad things happen when you
do this.  We (think we) have sufficient mechanisms in glibc to avoid
needing to do this, for instance symbol versioning.

So we need a libio in glibc which will work for _both_ the old ABI and
the new ABI.

Maybe we don't need it right now; for testing purposes, it may not
matter that the libio in libc and libstdc++ don't interoperate.  In
the long run, though---and that means before -fnew-abi becomes the
default in any release---we will need a libio that works for both libc
and libstdc++.  We will need to make a glibc release with this libio
at the same time as, or preferably before, the gcc release which
switches to the new ABI.  Given the lead times involved, you will
really want to be working on this now if you want the switch to happen
before 2001.

> The entirety of a single C++ program must be compiled using -fnew-abi,
> or -fno-new-abi -- you can't compile bits and pieces one way.  If you
> want to have a library that can be linked with the old ABI and the new
> ABI, you compile it twice: once to support the old ABI and once to
> support the new one.

No, this gives you two libraries.  You have to do something more
complicated if you only want one library.  Hopefully the new C++
symbol mangling and the old one don't overlap so the 'more
complicated' just means specifying both versions of the object files
on the link line.

> That's no different for libio, since it is emulating the C++ ABI, even
> though it is written in C.

libio is special because it is part of glibc.  The C++ ABI changes at
pretty frequent intervals, the C ABI doesn't.

>     >> It would be nice if people can use the new ABI, if only to test
>     >> it, on IA32 Linux.  They can't do that unless we can produce a
>     >> version of libio that doesn't interact with the C library,
>     >> since we can't expect everyone to go upgrading their C library.
> 
>     Geoff> Why not?  If the new ABI is stable (or stable enough for
>     Geoff> libio), there is no reason not to simply tell people "gcc
>     Geoff> version foo requires glibc version bar".
> 
> That's not very polite.  That means people with legacy Linux systems
> can't use the advances in the compiler.  But, what you say is
> possible.

This is not a new thing.  People with legacy Linux systems can't use
advances in the kernel unless they upgrade their libc, nor can they
run programs that were built under newer versions of the libc.  In
this case, they just won't be able to use the new ABI until they
upgrade, which probably won't cause them much pain.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Linus Torvalds]

In article < 199912202011.MAA12922@atrus.synopsys.com >,
Joe Buck  <jbuck@synopsys.COM> wrote:
>
>> In a pinch we can put in a temporary kludge, but only if it is
>> clearly marked as such.
>
>This may be an example of such a pinch.  The new ABI has to be tested.
>Requiring all testers to rebuild their glibc means that there will be
>a lot less testing, meaning a lower-quality release.

Ok, I'm probably being dense here, but why all these horrible problems
with backwards compatibility?

The whole _point_ of open source is to not have to play the same old
games all the time.

Why can't you just use a different version number for the test-library,
and then the new ABI doesn't impact the old one at all. Problem solved.

What's the reason for trying to have everything in one huge library?

		Linus

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912201435.PAA01388@loewis.home.cs.tu-berlin.de >you write:
  > It seems there is some confusion about the scope of what Mark attempts
  > to do.
  > 
  > He is working, in private, on implementing the new C++ ABI. Because he
  > is lacking an IA64 machine, he uses his private ia32 machine to do so.
I understand this.

  > To get it working at all, he must change libio, and he wants to avoid
  > rebuilding his Linux distribution to do so. I cannot find anything
  > unwise here.
He's more than welcome to do this on his local box.  But it's not something
that belongs in our source tree.

  > Please, can we leave the decision what to work on to the individual
  > developer, and only judge the quality of the work when we see actual
  > patches?
It's not an issue of quality, but of what belongs in our source tree.  I
still see no reason for this code to go in our source tree.

jeff

Re: Linux vs. libio

[Per Bothner]

Jamie Lokier <jamie.lokier@cern.ch> writes:

> The down side is it's not at all portable, so code that's written on
> Linux will output differently on other platforms.

Firstly, is not a downside if glibc provides well-defined expected
behavior where other platforms don't.  Secondly, the C++ standard
*does* require the standard streams cin/cout/cerr to be syncronized
with stdin/stdout/stderr.  You can do this portably, but you do an
extra useless plus a function call on every character.

> Only the /very/ few fields that are used by inline input/output
> functions need to be shared. Since they are simple data fields, sharing
> those would be very easy even with many different kinds of C++ ABI.
> 
> Everything else, including vtable format and placement does not need to
> be shared.  Just have the C library call functions (not virtual ones)
> like "flush_my_output_stream" etc., and those functions would contain
> the vtable lookup code.

I believe we do that.  At least we used to.  The intent was to partly
insulate user code from ABI changes.  However, you still need to
co-ordinate the C and C++ data structures.  If you do that, it seems
logical to *share* the data structures.  Otherwise, you have to use
extra indirection.  (You have to share the pointer(s) for the current
input/output position:  The same actual pointers have to be logically
in both the FILE and the streambuf.  So either one of these data
structures has to use an extra indirction, or they have to be the
same object, or at least the objects have to overlap.)
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    >> To get it working at all, he must change libio, and he wants to
    >> avoid rebuilding his Linux distribution to do so. I cannot find
    >> anything unwise here.

    Jeffrey> He's more than welcome to do this on his local box.  But
    Jeffrey> it's not something that belongs in our source tree.

Yes, it does.  There are three distinct things happenning here:

  o Implementation of -fnew-abi.

    We all agree this is a Good Thing.  Which platforms use it by
    default, when they do so, etc., is not something I've got any
    feelings about.  But, we want the new ABI to exist, and it will
    be the default ABI on IA64, I assume, since that's the ABI
    specified there.

    So, this goes in the tree.

  o Adjustments to libio.

    The current libio *will not work* without modification with the
    new ABI, even on non-Linux systems.  Suppose you're running 
    IRIX, and suppose you want to use the new ABI.  Or suppose
    you're running IA64 and you have to use the new ABI.  You need
    a modified libio.
    
    Of course, we need to conditionally compile the changes so that
    old ABI systems (like RH Linux) still work.  But, we can't
    support the new ABI without these changes, so they need to 
    go in the tree.

  o Configury changes to that building with -fnew-abi, on a Linux
    sytem using a glibc that contains a libio built with the old
    ABI, leaves you with a working compiler.

    If someone configures their tree using -fnew-abi on, say,
    RH 6.x, they'll get a totally non-working libio unless we
    turn on the new ABI in libio.  If that conflicts with the C
    library, and we don't sort out that interaction, they'll be hosed.

    There's no reason to say that people on IA32 Linux can't use
    the new ABI.  Just because libio happens to be in libc shouldn't
    (abstractly) sentence them to a worse fate than people running
    IRIX (where libio is not in libc).  The libio library knows how
    to deal with a "foreign" libc, and it should be possible to treat
    glibc as "foreign".  It may or may not be technically feasible
    to achieve this.

    I would argue that if it is achievable, this hunk should go in the
    tree as well.

I know there may be adverse consequences on performance as a result,
and I fully agree that the desirable configuration is to have a libc
and libstdc++ that share a libio.  But, for non-glibc systems we need
a solution, and for IA32 Linux I don't see why we shouldn't support
-fnew-abi configurations, if that's possible.

Believe it or not, I am not a *complete* idiot.  I'm not going to
somehow go breaking every Linux system on the planet, or something.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> Still, neither you nor I can answer my procedural question. :-)
    >> It's a steering committe issue: they (poor things!), not us,
    >> get to decide these questions. :-)

    Ulrich> Not all code in egcs is governed by the steering
    Ulrich> committee.  Just think about texinfo.  Simply because code
    Ulrich> was added to egcs for convenience reasons does not make
    Ulrich> the SC the owner.  It can decide about the future of that
    Ulrich> copy (= branch) but we all certainly want to avoid
    Ulrich> branches.

The FSF is the "owner" of all of this software.  :-)

But, I know what you mean.

It's true that the master version of texinfo is not in GCC.  It's also
true that I would, if for some reason the need arose, feel it was
reasonable for me to modify the version of texinfo in GCC, without
contacting the texinfo maintainers first.  Of course, if the changes
were generally useful, we'd submit them back to the texinfo folks as
well.

My understanding is that the SC *does* get to decide what to do about
stuff in the GCC tree.  That doesn't mean it has anything to say about
the *master* versions of those packages that are officially maintained
elsewhere.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mike Stump]

> Date: Tue, 21 Dec 1999 14:24:19 -0500 (EST)
> From: Kevin Atkinson <kevinatk@home.com>

> On 21 Dec 1999, Per Bothner wrote:

> Will that extra level of indirection *really* make a diffrence in
> speed I mean IO is something which is fairly slow in the first
> place.

It's always fun to hear from people that gzip is xx% slower when
compiled with the a new gcc, as compared to an older gcc.  Apparently
some programs that lots of people use, do IO, and people expect them
to be fast.

Anyway...

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

My guess is it can make a big difference.  The memory cost of copying
bytes around seems to be a significant factor in I/O.  At least the
people who work on kernels seem to think so, as they spend a lot of
effort on sharing buffers, copy-on-write, minimizing copying, DMA,
etc.

Whether the extra indirection for putc/getc type operations make
a real difference depends on how common they are compared to
other operations.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Per Bothner]

Joern Rennecke <amylaar@cygnus.co.uk> writes:

> C++ was known to be volatile when libio was written.
> The compatibility issues we have now could have been avoided if the API
> of libio had been designed to use plain C.

Duh.  libio defines some *C++* classes, whose specification
was (partially) specified by existing (de facto) standards.
Using plain C misses the point.  Libio was written to facilitate
integration of C and C++ I/O, which is *required* by the C++ standard.

There is (as always) a design tradeoff.  As I see it,
unified C/C++ I/O gives us:
+ Improved performance.
+ Conceptual elegance of uniting the I/O facilities of C/C++.
+ Easier mixing of C and C++ code (a GNU extension).
+ Sharing some tricky code (e.g. buffer management).

On the other hand, it costs us:
- Extra "hair".
- Compatibility issues made more complicated when two ABIs
  depend so closely on each other.

One can rationally argue for or against the unification.
An argument can be made that unification is "doing the right
thing" and well within the Gnu tradition of sacrificing
implementation complication for better performance or functionality.
What gets really annoying when C programmers make judgemental
statements without understanding the issues.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Gerald Pfeifer]

On Wed, 22 Dec 1999, Marc Espie wrote:
> Actually, I'm concerned about gcc and the steering committee position here.

Please note that the SC officially has not taken any position concerning
this issue.

Indeed the SC does not (and should) not act on all issues arising on the
lists, that's what we have all those maintainers and the release manager
(Jeff Law for the 2.95.x branch), all of which we trust, for. ;-)

Now, if there is serious disagreement among "global write" maintainers
which cannot be resolved by means of discussion, of course the SC will
take care of that. As a matter of fact, the SC is definitely not idle,
but where possible, decisions should be made by open discussion on the
list and consensus.

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> The libio tricks whereby there are C versions of C++ objects (complete
> with vtables, etc.) will still work, but the objects are going to have
> to be laid out differently, because the C++ objects are being laid out
> differently.

This is still no reason to break compatibility.  If the virtual tables
require a new layout one can add new ones.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Geoff Keating]

> From: Joe Buck <jbuck@synopsys.COM>
> Date: Mon, 20 Dec 99 10:04:02 PST
> 
> Mark Mitchell writes:
> 
> > I assumed that "blanket write privs" applied to everything you get
> > when you do a `cvs co egcs', which includes libio and libstdc++.
> 
> My opinion:
> 
> Blanket write privs are for the purpose of checking things in to CVS.
> Actually doing a release is another matter: for anything that affects
> other projects, the Steering Committee would have to be involved,
> and the SC tries to run by consensus, meaning that if Ulrich Drepper
> were extremely unhappy about something, the SC wouldn't approve its
> release.

My opinion:

The primary purpose of snapshots are for use in the testing and
further development of glibc.  Therefore, it is _not_ a good idea to
break them for no particular reason.  "blanket write privs" only
includes the committing of patches that _work_ (or at least are
believed to work).

> But remember, we never promised that we wouldn't break things in
> *snapshots*.  That's too much of a constraint.

It seems pointless to put stuff in snapshots that will have to be
backed out before a release.

Might I suggest the following flow-chart for Linux GCC ABI changes:

1. Is this patch simply for my internal use only?

   No    Yes------> keep it in your local repository,
   |		    don't put it in CVS at all.  If you think
   |		    others are interested, send it to gcc-patches.
   V
2. Will this patch work on currently shipping Red Hat,
   Debian, Suse, whatever systems, both if you use a new gcc
   on these systems with existing libraries and if you use a new
   gcc to build the libraries but still use old binaries.

   No	 Yes ----->  Then it's probably OK for the mainline tree.
   |
   V
3. Is this patch so important, so useful, that it is acceptable
   to break backwards binary compatibility?

   Yes   No ------>  Then it can't go in, and must be reworked.
   |		     When it's reworked, go back to (2).
   V
4. Can I obtain the consensus of everyone working on or using
   GCC under Linux?

   No    Yes ----->  Budget 1-2 years for building the consensus.  
   |		     Then, once you have that consensus, it can
   |		     go in, at an agreed time, preferably
   |		     with a major version number change for gcc.
   |		     Note that any shared libraries affected
   |		     will also need to bump their version and
   |		     this needs to be coordinated.
   |		     This is a _lot_ of work (budget 50% of
   |		     an engineer/manager's time for the 1-2 years).
   V
5. Your answer at (3) was wrong.
   The patch can't go in, and must be reworked.
   When it's reworked, go back to (2).

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> If you want to update the version of libio in the compiler, by all
> means go ahead!

I would have done this if it would make sense.  But it does not since
the changes are not yet fully tested and probably not even complete.
Adding all the wide char stuff is non-trivial.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Kevin Atkinson]

Per Bothner wrote:

> Duh.  libio defines some *C++* classes, whose specification
> was (partially) specified by existing (de facto) standards.
> Using plain C misses the point.  Libio was written to facilitate
> integration of C and C++ I/O, which is *required* by the C++ standard.
> 
> There is (as always) a design tradeoff.  As I see it,
> unified C/C++ I/O gives us:
> + Improved performance.
> + Conceptual elegance of uniting the I/O facilities of C/C++.
> + Easier mixing of C and C++ code (a GNU extension).
> + Sharing some tricky code (e.g. buffer management).
> 
> On the other hand, it costs us:
> - Extra "hair".
> - Compatibility issues made more complicated when two ABIs
>   depend so closely on each other.

Maybe I am missing something but couldn't the io code in linux be
written in pure C with a few extensions to allow C++ wrapper classes to
effectively use the io services.  Like say provide inline functions (or
macros) to manipulate the buffer efficiently.

-- 
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> things for users.  I think that the glibc folks should
    Joe> approve patches in the libraries with ABI-visible effects
    Joe> before they go to release branches, but we shouldn't need to
    Joe> wait for such approval to check something into the CVS
    Joe> archive.  After all, some problems won't be detected until
    Joe> people test snapshots anyway.

Thanks for clarifying!

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jason Merrill]

>>>>> Jeffrey A Law <law@cygnus.com> writes:

 > You will need to work with the glibc folks to find a solution that makes
 > sense to them as we do not want to start changing the API of the libio
 > contained in GCC without making corresponding changes to the libio
 > contained in glibc.

The problem is that part of the libio API is the C++ vtables, so any ABI
change that modifies the layout of the vtable (such as -fvtable-thunks) or
how the vtable is found breaks libio.  This has caused us a lot of trouble
over the years; perhaps the answer is to change that design, but this *is*
supposed to be the all-singing, all-dancing, ultimate C++ ABI.

Jason

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991221155340.P14133@namaste.lokigames-lan.com >you write:
  > > Somehow, I don't quite believe that, no matter how well-meaning the glibc
  > > folks have been, such disasters won't happen again...  
  > 
  > Neither do I. Sure, I'd love for the C++ ABI to settle, but I'm not
  > counting on it.
At some point it really does need to settle.  IMHO, the biggest reason it
hasn't settled is the major vendors have all been targetting their own
cpus (sparc, parisc, mips, powerpc, etc) and as a result there hasn't
been much interest in ABI work to allow code built with one vendor's tools
to interoperate with code built by another vendor's tools.

That's starting to change with ia64.  Suddenly a number of players want to
have some level of binary compatibility with each others tools, so of
course they have start defining an ABI.

It's been a long time coming, but I'm happy to see progress made.

  > > linux folks (like having to upgrade the glibc version number...), will th
  > > Steering Committee take steps to ensure the Right Thing is done ?  
  > 
  > I'm concerned about this, too.
Likewise.

  > I develop binary only software for Linux. It is a pain
  > (__register_frame_info, glibc 2.0 vs 2.1 ABI breakage, etc.).
Yes.  These have all been a major pain in the (*&@#$.  I'm not at all
happy with how these issues have been handled to date.

  > First and foremost, I want a compiler that works. If it's possible to
  > do that within the framework of backwards compatability, so much the
  > better--as long as it isn't a burden on further development.
Certainly we all want a compiler that works.  No doubt.  But if we're
going to be taken seriously we also have to spend a lot more time
dealing with backwards compatibility than we have in the past.   In fact,
IMHO, it was a lack of attention to backwards compatibility and
communication between gcc & glibc that lead to the ongoing 
__register_frame_info problems!

  > then they do. I expected things to break from libc5 to libc6. No
  > pointing whining about it. I did not expect things to break from glibc
  > 2.0 to 2.1. Perhaps that was my misunderstanding.
This is largely out of the control of the GCC project since we do not
control glibc.


jeff

Re: Linux vs. libio

[Gerald Pfeifer]

On Wed, 22 Dec 1999, Marc Espie wrote:
> Actually, I'm concerned about gcc and the steering committee position here.

Please note that the SC officially has not taken any position concerning
this issue.

Indeed the SC does not (and should) not act on all issues arising on the
lists, that's what we have all those maintainers and the release manager
(Jeff Law for the 2.95.x branch), all of which we trust, for. ;-)

Now, if there is serious disagreement among "global write" maintainers
which cannot be resolved by means of discussion, of course the SC will
take care of that. As a matter of fact, the SC is definitely not idle,
but where possible, decisions should be made by open discussion on the
list and consensus.

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: Linux vs. libio

[Linus Torvalds]

In article < 19991221155340.P14133@namaste.lokigames-lan.com >,
Michael Vance  <briareos@lokigames.com> wrote:
>
>What does bother me is when we agree that things won't break, and
>then they do. I expected things to break from libc5 to libc6. No
>pointing whining about it. I did not expect things to break from glibc
>2.0 to 2.1. Perhaps that was my misunderstanding.

Excellent point.

I personally disagree completely with the glibc approach of "backwards
comaptibility at all costs". 

It is _much_ preferable to be clearly new and different than to try to
be 100% compatible and succeeding to 99.9%. 

If there are major C++ internal ABI changes in gcc, glibc should just be
upgraded to the next major version, and all backwards compatibility
cruft should be removed.  People still have the old libraries installed,
and every single old binary works fine - simply because nothing has
changed as far as they know or care. 

Why are people discussing all these clever dual-vtable things at all?
Talk about subtle bugs..

I'm personally quite ashamed that people point to "Linux" as holding
things back (and I agree with them). Why is gcc so tightly linked to
glibc issues, and why can't glibc just change?

		Linus

Re: Linux vs. libio

[Joe Buck]

> I'd like reassurance on the following point: assuming there is some change
> in gcc that is necessary for the larger good of gcc (specifically, a modern
> C++ ABI, without vtables/thunks related bugs, and liable to work correctly
> from the start on IA64), that may mean some unpleasantness for glibc or
> linux folks (like having to upgrade the glibc version number...), will the
> Steering Committee take steps to ensure the Right Thing is done ?  

IMHO, the decision is up to the glibc people.  By that I mean that we've
already more or less committed that GCC 3.0 will by default support the
new all-singing all-dancing good-until-a-new-committee-breaks-things
C++ ABI.  The glibc maintainers can either preserve the same version
number by using whatever hooks are necessary, or upgrade the version
number.  If they can do the former without significant performance
penalties, then that will be the preferable choice.

BTW, how are the BSD folks handling the C++ requirements for consistency
between stdout/cout and stderr/cerr?

Re: Linux vs. libio

[Michael Vance]

On Wed, Dec 22, 1999 at 12:03:21AM +0100, Marc Espie wrote:

> I've followed the glibc issues from the sidelines.

As have I. I should note, for fairness sake, that I am a customer of
Mark's.

> Somehow, I don't quite believe that, no matter how well-meaning the glibc
> folks have been, such disasters won't happen again...  

Neither do I. Sure, I'd love for the C++ ABI to settle, but I'm not
counting on it.

> linux folks (like having to upgrade the glibc version number...), will the
> Steering Committee take steps to ensure the Right Thing is done ?  

I'm concerned about this, too.

I develop binary only software for Linux. It is a pain
(__register_frame_info, glibc 2.0 vs 2.1 ABI breakage, etc.). But, it is
not nearly as much a pain as having to track down very weird bugs in
the compiler (the atexit() crap for static object destructors in
shared libraries). Or having to find/pay someone who is
qualified/interested/etc. to do the work on the compiler. First and
foremost, I want a compiler that works. If it's possible to do that
within the framework of backwards compatability, so much the
better--as long as it isn't a burden on further development.

What does bother me is when we agree that things won't break, and
then they do. I expected things to break from libc5 to libc6. No
pointing whining about it. I did not expect things to break from glibc
2.0 to 2.1. Perhaps that was my misunderstanding.

> * companies that distribute binary-only applications.

As above, it doesn't help us if the compiler is broken.

m.

-- 
Programmer                         "I wrote a song about dental floss,
Loki Entertainment Software         but did anyone's teeth get cleaner?"
http://lokigames.com/~briareos/              - Frank Zappa, re: the PMRC

Re: Linux vs. libio

[Mike Stump]

> Date: Tue, 21 Dec 1999 14:24:19 -0500 (EST)
> From: Kevin Atkinson <kevinatk@home.com>

> On 21 Dec 1999, Per Bothner wrote:

> Will that extra level of indirection *really* make a diffrence in
> speed I mean IO is something which is fairly slow in the first
> place.

It's always fun to hear from people that gzip is xx% slower when
compiled with the a new gcc, as compared to an older gcc.  Apparently
some programs that lots of people use, do IO, and people expect them
to be fast.

Anyway...

Re: Linux vs. libio

[Marc Espie]

In article < Pine.LNX.4.10.9912211422460.7718-100000@kevins-linux >,
Kevin Atkinson <kevinatk@home.com> wrote:
>On 21 Dec 1999, Per Bothner wrote:

>> Kevin Atkinson <kevinatk@home.com> writes:

>> > >From http://www.sgi.com/Technology/STL/standard_library.html:

>> > Which is what I was trying to get at.  By directly maniplulating FILE
>> > one avoids the extra overhead.

>> Wise man say: "Every problem in computer science can be solved
>> by adding an extra level of indirection."   While SGI's technique
>> is clever, it is basically just another layer of indirection.
>> Specifically, you get at least two extra memory de-references
>> even in the fast case.

>Will that extra level of indirection *really* make a diffrence in speed I
>mean IO is something which is fairly slow in the first place.

It does make a *real* difference.  
I have some hard measurement data where I shaved 20% off a program's
execution time by getting data line by line and managing the buffer myself
instead of using fgetc.

Re: Linux vs. libio

[Marc Espie]

In article < 199912210311.TAA11055@localhost.cygnus.com > you write:
>> From: Mark Mitchell <mark@codesourcery.com>
>> Date: Mon, 20 Dec 1999 18:06:29 -0800
>
>>     Geoff> So we need a libio in glibc which will work for _both_ the
>>     Geoff> old ABI and the new ABI.
>> 
>> If we can do this, that will be wonderful.  I'm in no way opposed to
>> this, if it is technically feasible.  I'll try to see how this might
>> be done.
>...
>> But, one of the vtables will contain thunks that adjust the this
>> pointer before calling through the real vtable.  So there will be two
>> costs to just this level of compatibility: extra space, and extra
>> time.  That seems undesirable to me.

>It's not like there is a _choice_.  We must maintain backwards
>compatibility.  If this costs in space/time, then that's unfortunate
>but we have to live with it (and try to minimize it, of course).


This makes for yet another good reason to avoid linux and glibc.

Rude ? Maybe.

I've followed the glibc issues from the sidelines.
Somehow, I don't quite believe that, no matter how well-meaning the glibc
folks have been, such disasters won't happen again...  


Actually, I'm concerned about gcc and the steering committee position here.
I can understand that other FSF projects have a somewhat important influence
on gcc's life.

*However*, not all the world is FSF, there are other major users of gcc
outside of the glibc/linux worlds.

I'd like reassurance on the following point: assuming there is some change
in gcc that is necessary for the larger good of gcc (specifically, a modern
C++ ABI, without vtables/thunks related bugs, and liable to work correctly
from the start on IA64), that may mean some unpleasantness for glibc or
linux folks (like having to upgrade the glibc version number...), will the
Steering Committee take steps to ensure the Right Thing is done ?  


Furthermore, if you believe you will be able to live forever with glibc having
the same major version number, I think you're going down a very dangerous
path.

Updates happen.  Binary compatibility is not forever.  Update paths should be
provided, but the show must go on, problems must be fixed, and sooner or
later, there is going to be no other possibility than break old stuff (or get
more bloat than Windows).  As far as I understand, symbol versioning will
get you bloated systems/bloated libraries... ultimately a bad thing.
Whom does this help ?
* people for whom the update is too difficult,
* companies that distribute binary-only applications.

Both problems have real solutions: give people better tools to manage updates,
so that it becomes realistic for them to do so (notice how Windows makes 
this automatic... even though I loathe windows), 
and *don't* cater to companies that need such fine-grained binary 
compatibility. They can either put updates out, or start playing open, 
and work in a more system-frienly way.

Re: Linux vs. libio

[Geoff Keating]

> Date: Tue, 21 Dec 1999 14:24:19 -0500 (EST)
> From: Kevin Atkinson <kevinatk@home.com>

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

On my machine, I/O from gas and gcc is compute bound.  I expect it's
the same on most systems.  Modern caching techniques are pretty good,
and there aren't really many programs which have a working set of more
than 100Mb while compiling (and then it's usually because GCC is being
a glutton, not because of the size of the sources and objects).

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Ulrich Drepper]

Kevin Atkinson <kevinatk@home.com> writes:

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

No, it's not slow.  What Per is talking about are functions like
fputc, fgetc, also fputs, fgets etc.  These work on buffers and
normally don't include file operations.  And so it is a big handicap
to have extra indirection.

And something which wasn't mentioned: this methods requires a lot of
porting to each new platform (just look at how many platforms SGI's
stuff is working) and, perhaps more important, it is very risky.  The
libc is free to change the internals of the FILE structure since it is
an opaque type.  In glibc 2.2, for example, there will be support for
wide characters.  And the relations between the different buffers in
this context is really complicated.

So, manipulating the pointers in the FILE struct is not only slower,
it's also far less portable.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> Will that extra level of indirection *really* make a diffrence in speed I
> mean IO is something which is fairly slow in the first place.

My guess is it can make a big difference.  The memory cost of copying
bytes around seems to be a significant factor in I/O.  At least the
people who work on kernels seem to think so, as they spend a lot of
effort on sharing buffers, copy-on-write, minimizing copying, DMA,
etc.

Whether the extra indirection for putc/getc type operations make
a real difference depends on how common they are compared to
other operations.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Kevin Atkinson]

On 21 Dec 1999, Per Bothner wrote:

> Kevin Atkinson <kevinatk@home.com> writes:
> 
> > >From http://www.sgi.com/Technology/STL/standard_library.html:
> > ...
> > Which is what I was trying to get at.  By directly maniplulating FILE
> > one avoids the extra overhead.
> 
> Wise man say: "Every problem in computer science can be solved
> by adding an extra level of indirection."   While SGI's technique
> is clever, it is basically just another layer of indirection.
> Specifically, you get at least two extra memory de-references
> even in the fast case.

Will that extra level of indirection *really* make a diffrence in speed I
mean IO is something which is fairly slow in the first place.

--
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> >From http://www.sgi.com/Technology/STL/standard_library.html:
> ...
> Which is what I was trying to get at.  By directly maniplulating FILE
> one avoids the extra overhead.

Wise man say: "Every problem in computer science can be solved
by adding an extra level of indirection."   While SGI's technique
is clever, it is basically just another layer of indirection.
Specifically, you get at least two extra memory de-references
even in the fast case.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Marc Lehmann]

On Mon, Dec 20, 1999 at 03:29:27PM -0800, Mark Mitchell <mark@codesourcery.com> wrote:
>     Jeffrey> Not a bad idea.  On a branch would be fine by me.  We
>     Jeffrey> have used that scheme for similar purposes in the past.
> 
> If that's what the steering committee, or whoever decides this kind of
> thing, wants to do, then that's what we'll do.
> 
> But, I think it's really not a good idea.
> 
> In practice, this will lead to a lot less testing.
> 
> Doing the new ABI work has already spotted bugs in the compiler, and
> requires various other cleanups.  We'll lose those advantages in the
> mainline, and there will be semi-serious divergence.

Frankly, I do not understand these arguments.

While I am in favour of including this work in cvs, the idea of a branch
seems very reasonable for me.

If you wanted, you could make -fnew-abi the default there, so, instead of

  "for testing, you compile with -fnew-abi"

you do this:

  "for testing, check out -rlibio_new_abi"

My main problems with your reasoning is that you say, on one side: "all
code will be #ifdef'd out", but on the other side: "a branch will not be
tested as much" and "we might even be able to detect errors".

I think this needs clarification. For me, none of your arguments against a
branch seem to have substance.

Many similar projects have started as a branch first, and are now
integrated in the mainline sources.

Doing it on a branch (but in cvs) is as much open development as it being
on the main trunk. And it is a *major* help for the other maintainers (and
for the release manager(s)), in that it isolates the (substancial) patch
from the rest.

I do think you could just as well live with a branch, at least for some
time, until a solution for the libio problem is found.

And we do appreciate your work!

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@opengroup.org |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> If you want I'll ask for a formal vote from the steering
    Jeffrey> committee.

No, that will not be necessary.
 
We've discussed this internally at some length.  Since we're not
obligated to do the libio work at all (that was merely volunteerism on
our part, for the sake of the community at large), we've elected not
to do it.  

We'll just stick to our job: implement the new ABI support.  We'll do
that on a branch.  I belive it would be inappropriate to do the
-fnew-abi work on the mainline since that will be an incomplete
change: changing -fnew-abi and not changing libio will result in a
libstdc++ that does not work with -fnew-abi.  Of course, we will make
reasonable efforst to ensure that generic changes that come about as
part of this work appear in the mainline as well as on the branch.

It is our hope that some entity (perhaps CodeSourcery, if a customer
requests it) will eventually do the libio work, as well as the work
required to merge the -fnew-abi changes into the mainline.

I thank all of you for your time, ideas, comments and criticisms, and
I hope that the archive of this debate will assist the next person who
tackles this problem in formulating an elegant solution to the various
libio issues.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Kevin Atkinson]

Per Bothner wrote:

> One can write a C++ wrapper class (say stdiobuf) that implements the
> C++ functionality, but uses the C stdio buffers.  The problem is that
> this is slow, since you have to call handle_slow_case on every
> character.  You can force that by setting read_pointer == read_limit
> (for those streams that are aliased to stdio).  Alternatively, you can
> make sure to use the same value of read_pointer for C FILEs and C++
> filebufs.  However, then either you get extra overhead (indirection)
> or you have to merge the data structures for C and C++.

From http://www.sgi.com/Technology/STL/standard_library.html:

Another place where we've got system dependent code is syncing with
stdio. The C++ standard requires that there be a mode in which
iostreams and stdio are synchronized (the C++ standard is imprecise
about just what synchronization means, but, after talking to the
people who wrote that part of the standard, it's clear that it means a
very close coupling), and it requires that synchronized mode be the
default. We could have satisfied this requirement by implementing
things in terms of putc and getc, but that would have been horribly
inefficient. Instead we did something uglier, but more efficient: we
directly manipulate the pointers inside a FILE object. We've
encapsulated knowledge of struct FILE into the header
<stl_stdio_file.h>, which defines a few inline functions for this
manipulation. Again, this header has to be modified for every OS that
we target. This has not proven to be a serious problem in practice,
since most stdio implementations appear to be very similar to each
other.

Which is what I was trying to get at.  By directly maniplulating FILE
one avoids the extra overhead.

-- 
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Ulrich Drepper]

Per Bothner <per@bothner.com> writes:

> So whether "vtable compatibility" is an issue or not depends
> on which extended glibc-specific features are a documented
> part of the API and how programs access these features.

The problem is not the glibc code itself.  You are right, we can
change the vtables and it should not matter since none of these things
are documented.

The problem is passing C++ iostream classes to the functions in genops
and fileops.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Per Bothner]

Mark Mitchell <mark@codesourcery.com> writes:

> As was suggested, for example, the vtable pointer issue can probably
> be solved by having *two* vtable pointers in the new objects -- the
> one the C++ ABI requires, and the one the libio routines expect.  I
> think that's the *only* way to solve that particular problem, and I
> expect there will be other problems as well.

Well, for C code, it is not even clear you need to maintain
compatibility in terms of vtables.  As far as I know, of the
public C header files (such as stdio.h or libio.h) only call
named function;  calling a "method" (a slot in a jump table
or vtable) is only done internally.

So whether "vtable compatibility" is an issue or not depends
on which extended glibc-specific features are a documented
part of the API and how programs access these features.
There is also a slight danger of programs that use undocumented
backdoor features and functions;  that is what testing (such as
re-building all of say Debian) is for.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> main ()
> {
> 	printf ("This should be line 1.\n");
> 	cout << "This should be line 2.\n";
> }
> 
> might not work as expected, because stdout and cout will not be
> synchronized.  But most things will work just fine.  It would be much
> the same as the situation on an OS with a proprietary libc: the
> C++ and C i/o code would be separate.

This is required to work according to the C++ standard, so it better
work on an OS with a proprietary libc.  However, it will work much less
efficiently.

What will stop working is:

        fprintf((FILE*) cout.rdbuf(), "This should be line 1.\n");

I certainly don't see many programs breaking because of this ...
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210029.QAA17146@atrus.synopsys.com >you write:
  > Mark Mitchell writes:
  > 
  > >   > If that's what the steering committee, or whoever decides this kind o
  > f
  > >   > thing, wants to do, then that's what we'll do.
  > 
  > Jeff writes:
  > > A branch is certainly the direction I'm leaning.
  > 
  > OK, how about this.
  > 
  > Step 1: Mark checks in the code as a branch.
  > 
  > Step 2: Mark works with Uli to find an patch that the glibc team can
  > accept, giving the *option* (not on by default) of separate
  > glibc/libstdc++ I/O structures.  The goal is to be able to run with
  > the production glibc and -fnew-abi for C++.
  > 
  > (glibc folks: please be open to those gcc developers and testers who want
  > to work with a stable glibc and an experimental libstdc++ on Linux.
  > Without it, we're never going to get the libstdc++ quality up.)
  > 
  > Step 3: Once agreement is obtained, the code is put into the main gcc branc
  > h.
Sounds reasonable to me.

  > >   >   o We're not going to do stuff willy-nilly without asking the libio
  > >   >     folks for approval.
  > > This is the issue.  And I've stated before that if you get a buy-in from
  > > the glibc folks then you can go forward.  But you need a buy in *before*
  > > you start checking in the changes.
  > 
  > Fine (if "checking in" means "checking in on the main branch").
Yes.  That is precisely what I meant.  I should have been clearer on that 
point.

jeff

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912210049.QAA17857@atrus.synopsys.com >you write:
  > I wrote:
  > 
  > > If glibc and libstdc++ are to share the same structures for streams/stdio
  > ,
  > > then yes, they must change in lock-step.  This is appropriate for
  > > *released* versions.
  > 
  > Jeff writes:
  > 
  > > And do you happen to know when the next release cycle is going to start? 
  >  Are
  > > you going to volunteer to remove this code if the release cycle starts be
  > fore
  > > glibc & gcc have merged their libio implementations?  Are you going to
  > > volunteer to merge the two versions if Mark & CodeSourcery ultimately do 
  > not
  > > do the work?
  > 
  > Is this rhetorical or do you mean it?  In case you do mean it:
  > If that extremely unlikely event occurred, I'd just ship the thing and no
  > one would care.
Both.  And people would care -- every time we've mucked around with the
internals of libio we've created problems.  I will not support installing
this kind of code into the development libio without the glibc folks getting
on board first.  Even if the code is only conditionally used.

libio is (in effect) an external set of sources that we just happen to keep
a copy of.  We should not be mucking around with it.  Doing so just makes
our life harder when we have to untangle the mess that making private changes
to libio creates.  I will not be a party to such stupidity.

Even if we think we've got the code 100% correct (ie the behavior only changes
for -fnew-abi), then I still consider it unacceptable to change libio without
prior consent from the libio folks.  No users's won't notice, but developers
that have to apply bugfixes, or bring in a new version of libio from glibc
will be affected in a negative way.  Merges suck in a major way.  We've
managed to kill the last need for a merge when we picked up the remaining
gcc2 changes.  Let's not introduce new merge requirements.

  > Nevertheless, we're in violent agreement: the way to proceed is to start
  > out with a separate branch, then work on patches that all can accept.
Right.  So are you and I agreed, get the code onto a branch to help facilitate
development until Mark & Uli can come to an agreement on how to handle the
libio changes?

jeff

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220163854A.mitchell@codesourcery.com >you write:
  > >>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:
  > 
  >     Jeffrey> IMHO, even for development versions Mark needs to get the
  >     Jeffrey> glibc folks on board first or that work needs to happen
  >     Jeffrey> on a branch.
  > 
  > There are reasons to get the new ABI code in, even without libio
  > changes.  Such as anyone using GCC on an IA64 system needs those
  > changes.  They can get another free C++ iostreams library, like the
  > one from SGI, and use it instead of libio. 
Sorry.  IMHO, you must get the glibc folks on board before you can check in
these changes to libio.

If you want I'll ask for a formal vote from the steering committee.

 >  the new ABI work (in the compiler, not libio) is just like
  > adding support for the MIPS R10K, or for a new generic optimization,
  > or for a new language feature -- as long as it's not impacting the
  > rest of the compiler, and it's a good thing to do, it belongs on the
  > mainline, right?  This isn't like old-IA32 vs. new-IA32, or
  > old-fixinclude vs. new-fixincludes; there will never be a cutover from
  > the old ABI to the new ABI.  Instead, they'll both coexist, with a
  > switch to choose which one to use, for the forseeable future.
I've got no objection to the compiler side of the ABI work going into the
development tree.  My objections are with such changes going into libio
without prior approval from the glibc/libio folks.  I thought I had made
that clear.

jeff

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 18:06:29 -0800

>     Geoff> So we need a libio in glibc which will work for _both_ the
>     Geoff> old ABI and the new ABI.
> 
> If we can do this, that will be wonderful.  I'm in no way opposed to
> this, if it is technically feasible.  I'll try to see how this might
> be done.
...
> But, one of the vtables will contain thunks that adjust the this
> pointer before calling through the real vtable.  So there will be two
> costs to just this level of compatibility: extra space, and extra
> time.  That seems undesirable to me.

It's not like there is a _choice_.  We must maintain backwards
compatibility.  If this costs in space/time, then that's unfortunate
but we have to live with it (and try to minimize it, of course).

If we did have to do this, we could at least try to make the runtime
cost apply only to old applications.  That way, it will diminish over
time.

> Can you suggest a way of making this work that doesn't involve a
> performance hit?

Another way to make it work is to have glibc detect the version of gcc
in use and have the new libio be the default when the new gcc is being
used, through the use of symbol versioning---that is, have two
completely separate copies of libio inside glibc "like multilibs, but
at runtime" (now _there's_ a slogan that inspires confidence :-) ).
Possibly something could be done to make this work without recompiling
glibc.  You couldn't mix C++ shared objects compiled with both
compilers, but I don't think this is a serious problem.

One advantage of this is that we could make the new-ABI code more
efficient, as it would avoid some of the backwards compatibility
baggage we have now.

>     Geoff> more complicated if you only want one library.  Hopefully
>     Geoff> the new C++ symbol mangling and the old one don't overlap
>     Geoff> so the 'more complicated' just means specifying both
>     Geoff> versions of the object files on the link line.
> 
> The mangling has not yet been decided, so that's unclear.  But note
> that, in your proposal, even the C parts of libio need to handle both
> ABIs, so I don't think mangling is enough to keep things totally
> separate.

I'm not sure that libio needs to handle both ABIs _in the same
process_.  Just inside the same libc.so.

>     Geoff> libio is special because it is part of glibc.  The C++ ABI
>     Geoff> changes at pretty frequent intervals, the C ABI doesn't.
> 
> It is the intention of myself (and I believe Jason Merrill would
> agree, after the new ABI is implemented, never to change it again --
> just as you folks have committed to with libc.  We recognize that one
> of the obstacles to widespread C++ acceptance has been the lack of a
> stable ABI, and I'm hoping that the new ABI, which has been developed
> by a committee of intelligent people, will be good enough that we can
> stick with it indefinitely.  Time will tell, of course.

Oh good!  I would very much support this.

> One of my concerns is testing: I want people to be able to test the
> new ABI.  Lots of those potential testers are on IA32 Linux, but they
> don't really want to upgrade their systems, or even have to install
> new versions of libc in funny hidden places just to do the testing.

For testing purposes, you can just say "check libio out from this
branch" and put whatever hacks you like in it; that's no problem.
But before the release we do need to solve the whole problem.

> But I think you and I will have to agree to disagree on this point; I
> think we just evaluate the likelihood of people being willing to
> upgrade differently.



-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> I wrote:

    >> Are you sure that you understand Mark's proposal?  I ask this
    >> because your assertion is incorrect -- there would be two
    >> separate libio's involved.  (This is not a new thing: we used
    >> tricks like these back in the libc5 days sometimes).

    Joe> Whoops, it appears that I was assuming that Mark was thinking
    Joe> of doing things this way, but now I see that this is not at
    Joe> all clear.  But yes, libio as originally written was designed
    Joe> to be used this way (either as both the C and the C++ I/O
    Joe> implementation, or as only the C++ implementation).

That was indeed my original intent.  The waters have become muddied at
this point, no doubt.

But, I think you have faithfully transcribed my original proposal, and
the one with which I feel the most comfort.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

I wrote:

> Are you sure that you understand Mark's proposal?  I ask this because
> your assertion is incorrect -- there would be two separate libio's
> involved.  (This is not a new thing: we used tricks like these back
> in the libc5 days sometimes).

Whoops, it appears that I was assuming that Mark was thinking of doing
things this way, but now I see that this is not at all clear.  But yes,
libio as originally written was designed to be used this way (either
as both the C and the C++ I/O implementation, or as only the C++
implementation).

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

> >   o I'm not proposing changing *anything* about libio when compiled
> >     with the existing C++ ABI, which will continue to be supported
> >     indefinitely.
> > 
> >   o I'm proposing chaning libio, conditionally, when compiled with the
> >     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
> >     so, it will not work.

Geoff Keating writes:
> This is what I'm objecting to.  Perhaps I should draw a diagram.  I
> expect that with -fnew-abi, the version number of libstdc++ will
> change, perhaps to libstdc++.so.3.  Now, suppose I use my brand new
> GCC 3.0.0, with -fnew-abi on by default, under my linux system,
> to compile an executable 'test', and 'netscape' is an old binary-only
> application compiled under the old GCC.
> 
> So we have a shared library use graph that looks like this:
> 
> netscape --> libstdc++.so.2.8 --> libc.so.6
> test	 --> libstdc++.so.3   ----^
> 
> But this won't work with your proposal. 

Are you sure that you understand Mark's proposal?  I ask this because
your assertion is incorrect -- there would be two separate libio's
involved.  (This is not a new thing: we used tricks like these back
in the libc5 days sometimes).

His conditionally changed libio would live inside libstdc++.so.3, and the
C bindings for it would not be exported (exactly as it does on, say,
Solaris platforms).  Suppose test does some C stdio stuff in addition to
the C++ iostreams stuff.  In that case, it would get the iostreams stuff
from libstdc++.so.3, and would get the stdio stuff from libc.so.6.  There
would be two copies of libio in the same address space, one exporting the
C++ bindings and one exporting the C bindings.  The main user-visible
symptom would be that programs like this:

#include <stdio.h>
#include <iostream.h>

main ()
{
	printf ("This should be line 1.\n");
	cout << "This should be line 2.\n";
}

might not work as expected, because stdout and cout will not be
synchronized.  But most things will work just fine.  It would be much
the same as the situation on an OS with a proprietary libc: the
C++ and C i/o code would be separate.

Now, we don't want this as a long-term solution.  But it doesn't have
severe consequences as a temporary solution to be used by developers
and testers.

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@cygnus.com> writes:

Thanks for your email.  You were very clear.

    Geoff> So we have a shared library use graph that looks like this:

    Geoff> netscape --> libstdc++.so.2.8 --> libc.so.6 test -->
    Geoff> libstdc++.so.3 ----^

    Geoff> But this won't work with your proposal.  I can have the
    Geoff> libio in libc.so.6 compiled with the new C++ ABI, in which
    Geoff> case the old libstdc++ won't work, or it can be compiled
    Geoff> with the old ABI in which case the new libstdc++ won't
    Geoff> work.

Right; that's why I was proposing figuring out an *optional* way of
divorcing libio from libstdc++ -- exactly to avoid this problem.  I
understand that's not your long-term goal, but I wanted to support
exactly this situation.  So, I envisioned all of:

  o libstdc++ with old ABI working with old libc.
  o libstdc++ with new ABI working with new libc.
  o libstdc++ with new ABI working with old libc.
  o libstdc++ with old ABI working with new libc.

    Geoff> So we need a libio in glibc which will work for _both_ the
    Geoff> old ABI and the new ABI.

If we can do this, that will be wonderful.  I'm in no way opposed to
this, if it is technically feasible.  I'll try to see how this might
be done.

In case you missed the earlier discussion, one of the issues is that
things like _IO_XSPUTN don't work because the vtable pointer is now
before the _IO_FILE, not after it.

As was suggested, for example, the vtable pointer issue can probably
be solved by having *two* vtable pointers in the new objects -- the
one the C++ ABI requires, and the one the libio routines expect.  I
think that's the *only* way to solve that particular problem, and I
expect there will be other problems as well.

But, one of the vtables will contain thunks that adjust the this
pointer before calling through the real vtable.  So there will be two
costs to just this level of compatibility: extra space, and extra
time.  That seems undesirable to me.

Can you suggest a way of making this work that doesn't involve a
performance hit?

    Geoff> more complicated if you only want one library.  Hopefully
    Geoff> the new C++ symbol mangling and the old one don't overlap
    Geoff> so the 'more complicated' just means specifying both
    Geoff> versions of the object files on the link line.

The mangling has not yet been decided, so that's unclear.  But note
that, in your proposal, even the C parts of libio need to handle both
ABIs, so I don't think mangling is enough to keep things totally
separate.

    Geoff> libio is special because it is part of glibc.  The C++ ABI
    Geoff> changes at pretty frequent intervals, the C ABI doesn't.

It is the intention of myself (and I believe Jason Merrill would
agree, after the new ABI is implemented, never to change it again --
just as you folks have committed to with libc.  We recognize that one
of the obstacles to widespread C++ acceptance has been the lack of a
stable ABI, and I'm hoping that the new ABI, which has been developed
by a committee of intelligent people, will be good enough that we can
stick with it indefinitely.  Time will tell, of course.

One of the things that has happenned up until now is that new features
of C++ have required ABI changes.  The new ABI does explicitly deal
with all the lanague features.  My primary fear is not inadequacies in
the ABI -- it is bugs in the ABI implementation in the compiler.

    Geoff> This is not a new thing.  People with legacy Linux systems
    Geoff> can't use advances in the kernel unless they upgrade their
    Geoff> libc, nor can they run programs that were built under newer
    Geoff> versions of the libc.  In this case, they just won't be
    Geoff> able to use the new ABI until they upgrade, which probably
    Geoff> won't cause them much pain.

One of my concerns is testing: I want people to be able to test the
new ABI.  Lots of those potential testers are on IA32 Linux, but they
don't really want to upgrade their systems, or even have to install
new versions of libc in funny hidden places just to do the testing.

But I think you and I will have to agree to disagree on this point; I
think we just evaluate the likelihood of people being willing to
upgrade differently.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Mon, 20 Dec 1999 15:04:59 -0800

>     Geoff> conditionally _compile_ anything.  You must make sure
>     Geoff> everything will work at runtime, out of the same objects.
>     Geoff> This is what 'binary compatibility' means.
> 
> Yes.  Let's try this again.
...
>   o On Linux, life is more difficult -- libstdc++ and glibc know
>     about each other.
> 
>     I was thinking that *in principle* one could set up libio
>     in libstdc++ analagously to the IRIX situation.  In particular,
>     without knowing glibc, just as we do on IRIX.  Of course, one
>     would only do this for the new ABI, and only when the system
>     libc doesn't already contain the new libio.
> 
>     It may be that this is technically infeasible.  It's certainly
>     not wrong in principle -- it's in fact the only way to let people
>     use the new ABI without upgrading their C library.

In principle, it would be better to have libio and IRIX libc know
about each other.  Unfortunately, this isn't practical.  Under GNU
operating systems (OSs using glibc) this is possible, and very
desirable.

> To quote Nixon :-), "let me make one thing perfectly clear":
> 
>   o I'm not proposing changing *anything* about libio when compiled
>     with the existing C++ ABI, which will continue to be supported
>     indefinitely.
> 
>   o I'm proposing chaning libio, conditionally, when compiled with the
>     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
>     so, it will not work.

This is what I'm objecting to.  Perhaps I should draw a diagram.  I
expect that with -fnew-abi, the version number of libstdc++ will
change, perhaps to libstdc++.so.3.  Now, suppose I use my brand new
GCC 3.0.0, with -fnew-abi on by default, under my linux system,
to compile an executable 'test', and 'netscape' is an old binary-only
application compiled under the old GCC.

So we have a shared library use graph that looks like this:

netscape --> libstdc++.so.2.8 --> libc.so.6
test	 --> libstdc++.so.3   ----^

But this won't work with your proposal.  I can have the libio in
libc.so.6 compiled with the new C++ ABI, in which case the old
libstdc++ won't work, or it can be compiled with the old ABI in
which case the new libstdc++ won't work.

It is the glibc maintainers' policy to never again change the libc
version number if we can possibly help it.  Bad things happen when you
do this.  We (think we) have sufficient mechanisms in glibc to avoid
needing to do this, for instance symbol versioning.

So we need a libio in glibc which will work for _both_ the old ABI and
the new ABI.

Maybe we don't need it right now; for testing purposes, it may not
matter that the libio in libc and libstdc++ don't interoperate.  In
the long run, though---and that means before -fnew-abi becomes the
default in any release---we will need a libio that works for both libc
and libstdc++.  We will need to make a glibc release with this libio
at the same time as, or preferably before, the gcc release which
switches to the new ABI.  Given the lead times involved, you will
really want to be working on this now if you want the switch to happen
before 2001.

> The entirety of a single C++ program must be compiled using -fnew-abi,
> or -fno-new-abi -- you can't compile bits and pieces one way.  If you
> want to have a library that can be linked with the old ABI and the new
> ABI, you compile it twice: once to support the old ABI and once to
> support the new one.

No, this gives you two libraries.  You have to do something more
complicated if you only want one library.  Hopefully the new C++
symbol mangling and the old one don't overlap so the 'more
complicated' just means specifying both versions of the object files
on the link line.

> That's no different for libio, since it is emulating the C++ ABI, even
> though it is written in C.

libio is special because it is part of glibc.  The C++ ABI changes at
pretty frequent intervals, the C ABI doesn't.

>     >> It would be nice if people can use the new ABI, if only to test
>     >> it, on IA32 Linux.  They can't do that unless we can produce a
>     >> version of libio that doesn't interact with the C library,
>     >> since we can't expect everyone to go upgrading their C library.
> 
>     Geoff> Why not?  If the new ABI is stable (or stable enough for
>     Geoff> libio), there is no reason not to simply tell people "gcc
>     Geoff> version foo requires glibc version bar".
> 
> That's not very polite.  That means people with legacy Linux systems
> can't use the advances in the compiler.  But, what you say is
> possible.

This is not a new thing.  People with legacy Linux systems can't use
advances in the kernel unless they upgrade their libc, nor can they
run programs that were built under newer versions of the libc.  In
this case, they just won't be able to use the new ABI until they
upgrade, which probably won't cause them much pain.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Joe Buck]

I wrote:

> If glibc and libstdc++ are to share the same structures for streams/stdio,
> then yes, they must change in lock-step.  This is appropriate for
> *released* versions.

Jeff writes:

> And do you happen to know when the next release cycle is going to start?  Are
> you going to volunteer to remove this code if the release cycle starts before
> glibc & gcc have merged their libio implementations?  Are you going to
> volunteer to merge the two versions if Mark & CodeSourcery ultimately do not
> do the work?

Is this rhetorical or do you mean it?  In case you do mean it:
If that extremely unlikely event occurred, I'd just ship the thing and no
one would care.

The reason no one would care is this: users who do not specify -fnew-abi
would not see any difference, as Mark's changes would be #ifdef'd out.
(Mark's proposal results in NO CHANGE unless libstdc++ is built with
-fnew-abi!).

People using -fnew-abi would notice that some programs that mix stdio
and iostreams don't work, and that more memory is consumed due to two
libio variant copies.  This would be suboptimal but OK for a non-default,
unfinished feature.

Nevertheless, we're in violent agreement: the way to proceed is to start
out with a separate branch, then work on patches that all can accept.

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> IMHO, even for development versions Mark needs to get the
    Jeffrey> glibc folks on board first or that work needs to happen
    Jeffrey> on a branch.

There are reasons to get the new ABI code in, even without libio
changes.  Such as anyone using GCC on an IA64 system needs those
changes.  They can get another free C++ iostreams library, like the
one from SGI, and use it instead of libio.  

But, they can't ship ABI-compliant code without an ABI-compliant C++
compiler.

Doing the new ABI work (in the compiler, not libio) is just like
adding support for the MIPS R10K, or for a new generic optimization,
or for a new language feature -- as long as it's not impacting the
rest of the compiler, and it's a good thing to do, it belongs on the
mainline, right?  This isn't like old-IA32 vs. new-IA32, or
old-fixinclude vs. new-fixincludes; there will never be a cutover from
the old ABI to the new ABI.  Instead, they'll both coexist, with a
switch to choose which one to use, for the forseeable future.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> That's the chance you and everyone must take when they
    Jeffrey> decide to work with free software projects.  If you don't
    Jeffrey> have the resources to do the merge, then you (in reality)
    Jeffrey> under-bid the work by under-estimating the integration
    Jeffrey> aspects of the contract. 

I don't think discussing these kinds of details on this list is
appropriate.  Our business practices are our business, just as Cygnus'
are yours.  They've no place here.

Integrating our changes is something we do voluntarily, as a service
to the community -- not because we have to.

I believe the proposal I made in my last email is reasonable, and I
believe it is the one whereby the community comes out the best: we
deliver a working libio and working new ABI, integrated on the
development branch of GCC, and the only cost is some temporary
instability in GCC when using -fnew-abi -- an already unsupported
option.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

>   > If that's what the steering committee, or whoever decides this kind of
>   > thing, wants to do, then that's what we'll do.

Jeff writes:
> A branch is certainly the direction I'm leaning.

OK, how about this.

Step 1: Mark checks in the code as a branch.

Step 2: Mark works with Uli to find an patch that the glibc team can
accept, giving the *option* (not on by default) of separate
glibc/libstdc++ I/O structures.  The goal is to be able to run with
the production glibc and -fnew-abi for C++.

(glibc folks: please be open to those gcc developers and testers who want
to work with a stable glibc and an experimental libstdc++ on Linux.
Without it, we're never going to get the libstdc++ quality up.)

Step 3: Once agreement is obtained, the code is put into the main gcc branch.

>   >   o We're not going to do stuff willy-nilly without asking the libio
>   >     folks for approval.
> This is the issue.  And I've stated before that if you get a buy-in from
> the glibc folks then you can go forward.  But you need a buy in *before*
> you start checking in the changes.

Fine (if "checking in" means "checking in on the main branch").

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912202336.PAA16416@atrus.synopsys.com >you write:
  > 
  > >   > It isn't really required that libio and glibc change in lockstep.  It
  > >   > certainly doesn't matter on platforms that don't use glibc.
  > 
  > > Huh?  If you don't change them in lockstep you introduce incompatibilitie
  > s
  > > between the version in the linux C library and the version in GCC.  That
  > > has proven to be a majorly stupid thing to do.  We don't want to repeat
  > > that mistake again.  So I'll repeat.  They must change in lock step.
  > 
  > If glibc and libstdc++ are to share the same structures for streams/stdio,
  > then yes, they must change in lock-step.  This is appropriate for
  > *released* versions.
And do you happen to know when the next release cycle is going to start?  Are
you going to volunteer to remove this code if the release cycle starts before
glibc & gcc have merged their libio implementations?  Are you going to
volunteer to merge the two versions if Mark & CodeSourcery ultimately do not
do the work?

IMHO, even for development versions Mark needs to get the glibc folks on board
first or that work needs to happen on a branch.


  > But for *testing*, it would be nice to have the
  > *option* on Linux of saying "don't share i/o structures".  Enabling this
  > option would break programs that rely on the C++ standard's guarantee that
  > cout/stdout and cerr/stderr are synchronized (unless flush calls are
  > inserted at appropriate points), and we would never build binary packages
  > to be distributed in that way, though we could point users to this choice
  > if they are stuck, trying to get new C++ code to run on a legacy Linux
  > system, for instance.
All this can be easily done on a branch.

  > Ideally, this option would not represent a fork because we could find
  > an agreed-apon way to do this (agreeable to both the C++ and glibc teams).
Yup.  If we can do this, then all my objections are removed.  Ultimately this
is what I want -- Mark, CodeSourcery, Uli & the glibc team to agree on
a technical plan and work together to make it happen and keep the glibc
and gcc sources in sync.  That *must* happen eventually anyway, or everything
Mark & CodeSourcery are doing will end up being tossed into the trash can.

  > It appears that the only way to get past this impasse is to find a libio
  > patch that the glibc folks would be willing to accept.
Right.  That's precisely what needs to happen short and long term.  It's
the right thing to do.  While they are working towards such a solution Mark
and CodeSourcery are free to work on a branch to expose the code to additional
testers.

jeff

Re: Linux vs. libio

[Linus Torvalds]

In article < 199912202011.MAA12922@atrus.synopsys.com >,
Joe Buck  <jbuck@synopsys.COM> wrote:
>
>> In a pinch we can put in a temporary kludge, but only if it is
>> clearly marked as such.
>
>This may be an example of such a pinch.  The new ABI has to be tested.
>Requiring all testers to rebuild their glibc means that there will be
>a lot less testing, meaning a lower-quality release.

Ok, I'm probably being dense here, but why all these horrible problems
with backwards compatibility?

The whole _point_ of open source is to not have to play the same old
games all the time.

Why can't you just use a different version number for the test-library,
and then the new ABI doesn't impact the old one at all. Problem solved.

What's the reason for trying to have everything in one huge library?

		Linus

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220152927Y.mitchell@codesourcery.com >you write:
  > If that's what the steering committee, or whoever decides this kind of
  > thing, wants to do, then that's what we'll do.
A branch is certainly the direction I'm leaning.

  > But, I think it's really not a good idea.
  > 
  > In practice, this will lead to a lot less testing.
But it will avoid diverging our libio sources unnecessarily, which reduces our
development/support burden.

You have no idea (nor does anyone else right now) when the next release cycle
will start.  We don't need the extra burden of forcing the release manager to
pull your code out of the mainline sources because it wasn't finished, or the
glibc folks decided to go a different direction, etc etc.

Until such time as you've got a buy-in from the glibc folks you need to either
keep the code private or work from a branch.

  > Doing the new ABI work has already spotted bugs in the compiler, and
  > requires various other cleanups.  We'll lose those advantages in the
  > mainline, and there will be semi-serious divergence.
Just like the new ia32 port, garbage collection and new fixinclues did.  But
ultimately doing it on a branch was the right thing to do in each of those
cases.  I believe we're in a similar situation now -- from a maintenance
standpoint (as opposed to a stability standpoint).


  > We (CodeSourcery) might not have the resources to do the merge which
  > might mean the community may be unable to profit from the new ABI
  > until someone volunteers to do it.
That's the chance you and everyone must take when they decide to work with
free software projects.  If you don't have the resources to do the merge, then
you (in reality) under-bid the work by under-estimating the integration 
aspects of the contract.  I problem I am unfortunately all too familiar with.


  >   Think how long the GC work done on
  > the branch by Bernd and Richard might have languished had we not
  > stepped in to integrate it.
Yes.  I'm well aware of that.  Again, that's the chance you have to take to
play the free software game.  Note that the GC work was done entirely without
a customer and thus had nothing to drive it to completion.


  >   o We're not going to break IA32 Linux.
This isn't the issue.

  >   o We're not going to do stuff willy-nilly without asking the libio
  >     folks for approval.
This is the issue.  And I've stated before that if you get a buy-in from
the glibc folks then you can go forward.  But you need a buy in *before*
you start checking in the changes.

  > What's happenning here is that we're doing open development.
And you can just as easily do open development on a branch.  You can even
use the branch for highly experimental work to get it to a point where you
and the glibc folks agree on the general direction.

jeff

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220150459I.mitchell@codesourcery.com >you write:
  > Let's suppose we're running on IRIX.  There's no libio in libc.
  > 
  >   o Now, we can compile with the old C++ ABI or the new C++ ABI.  
  >     Either is OK -- there's no standard C++ ABI on IRIX.  The user
  >     might choose this which ABI to use at compiler-configuration time.
  > 
  >     But, the libio in libstdc+++ needs to know about the C++ ABI
  >     (that's a design choice in libio).  So, we need to make the libio
  >     in libstdc++ do its by-hand layout of C++ objects in the same 
  >     way as the C++ compiler.  Which way to do this is known to us
  >     when we build libio; we've already committed to the C++ ABI to
  >     use.  Depending on which way the user configured the compiler,
  >     we'll end up with different versions of libio.
  >     
  >   o On Linux, life is more difficult -- libstdc++ and glibc know
  >     about each other.
  > 
  >     I was thinking that *in principle* one could set up libio
  >     in libstdc++ analagously to the IRIX situation.  In particular,
  >     without knowing glibc, just as we do on IRIX.  Of course, one
  >     would only do this for the new ABI, and only when the system
  >     libc doesn't already contain the new libio.
  > 
  >     It may be that this is technically infeasible.  It's certainly
  >     not wrong in principle -- it's in fact the only way to let people
  >     use the new ABI without upgrading their C library.
  > 
  > To quote Nixon :-), "let me make one thing perfectly clear":
  > 
  >   o I'm not proposing changing *anything* about libio when compiled
  >     with the existing C++ ABI, which will continue to be supported
  >     indefinitely.
  > 
  >   o I'm proposing chaning libio, conditionally, when compiled with the
  >     new C++ ABI.  (This is analagous to multilibbing.)  Without doing
  >     so, it will not work.
  > 
  > The entirety of a single C++ program must be compiled using -fnew-abi,
  > or -fno-new-abi -- you can't compile bits and pieces one way.  If you
  > want to have a library that can be linked with the old ABI and the new
  > ABI, you compile it twice: once to support the old ABI and once to
  > support the new one.
  > 
  > That's no different for libio, since it is emulating the C++ ABI, even
  > though it is written in C.
  > 
  >     >> It would be nice if people can use the new ABI, if only to test
  >     >> it, on IA32 Linux.  They can't do that unless we can produce a
  >     >> version of libio that doesn't interact with the C library,
  >     >> since we can't expect everyone to go upgrading their C library.
  > 
  >     Geoff> Why not?  If the new ABI is stable (or stable enough for
  >     Geoff> libio), there is no reason not to simply tell people "gcc
  >     Geoff> version foo requires glibc version bar".
  > 
  > That's not very polite.  That means people with legacy Linux systems
  > can't use the advances in the compiler.  But, what you say is
  > possible.
  > 
  > However, we *still* have to change libio for other, non-Linux systems
  > -- or else nobody can use the new ABI *anywhere*.
All this is fine and good *as long as you can get Uli and the glibc folks to
agree to the change*.  I'm not going to support diverging our libio sources
from the glibc libio sources.  Even in the development tree.  That ultimately
is a lose.

I'll repeat, if you can get Uli & friends to agree to the change, then it can
go into the development sources.  If not, then it must not go into the
development sources (it can of course still go on a branch).


jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> If glibc and libstdc++ are to share the same structures for
    Joe> streams/stdio, then yes, they must change in lock-step.  This
    Joe> is appropriate for *released* versions.  But for *testing*,
    Joe> it would be nice to have the *option* on Linux of saying
    Joe> "don't share i/o structures".  Enabling this option would
    Joe> break programs that rely on the C++ standard's guarantee that
    Joe> cout/stdout and cerr/stderr are synchronized (unless flush
    Joe> calls are inserted at appropriate points), and we would never
    Joe> build binary packages to be distributed in that way, though
    Joe> we could point users to this choice if they are stuck, trying
    Joe> to get new C++ code to run on a legacy Linux system, for
    Joe> instance.

Exactly!  I'm ecstatic that you intuited what I was trying to say!

    Joe> It appears that the only way to get past this impasse is to
    Joe> find a libio patch that the glibc folks would be willing to
    Joe> accept.

That's what we were working on when I asked for help yesterday.
Sadly, we haven't made much progress; I've been too busy in this email
debate! :-(

I guess the mistake was to try to do the design in this way.  Instead,
we can just complete the patch, and present the finished product for
comment, and, hopefully, approval.

Let's avoid branching, then.  How about this:

  o We'll work on -fnew-abi support in the compiler.  Checking in the
    changes may break -fnew-abi regression tests, because of libio
    problems, but people can still see the changes and test -fnew-abi
    in non I/O streams examples.

  o In parallel, we'll work on the libio patches.  We'll submit them,
    and try to get approval.  Then, we'll put them in libio, bringing
    things back to zero regression-test failures.

There's no risk: nobody is supposed to be using -fnew-abi for anything
serious right now, non-new-abi builds won't break, and we'll keep the
libios in sync.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

>   > It isn't really required that libio and glibc change in lockstep.  It
>   > certainly doesn't matter on platforms that don't use glibc.

> Huh?  If you don't change them in lockstep you introduce incompatibilities
> between the version in the linux C library and the version in GCC.  That
> has proven to be a majorly stupid thing to do.  We don't want to repeat
> that mistake again.  So I'll repeat.  They must change in lock step.

If glibc and libstdc++ are to share the same structures for streams/stdio,
then yes, they must change in lock-step.  This is appropriate for
*released* versions.  But for *testing*, it would be nice to have the
*option* on Linux of saying "don't share i/o structures".  Enabling this
option would break programs that rely on the C++ standard's guarantee that
cout/stdout and cerr/stderr are synchronized (unless flush calls are
inserted at appropriate points), and we would never build binary packages
to be distributed in that way, though we could point users to this choice
if they are stuck, trying to get new C++ code to run on a legacy Linux
system, for instance.

Ideally, this option would not represent a fork because we could find
an agreed-apon way to do this (agreeable to both the C++ and glibc teams).

>   > Why should we make it hard for people to test the new ABI?
> Why should we diverge from glibc?  That just makes our life more difficult.
> 
> You've got find another way which doesn't cause our libio to diverge from
> the one in glibc.  No ifs ands or buts about it.

It appears that the only way to get past this impasse is to find a libio
patch that the glibc folks would be willing to accept.

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> Not a bad idea.  On a branch would be fine by me.  We
    Jeffrey> have used that scheme for similar purposes in the past.

If that's what the steering committee, or whoever decides this kind of
thing, wants to do, then that's what we'll do.

But, I think it's really not a good idea.

In practice, this will lead to a lot less testing.

Doing the new ABI work has already spotted bugs in the compiler, and
requires various other cleanups.  We'll lose those advantages in the
mainline, and there will be semi-serious divergence.

We (CodeSourcery) might not have the resources to do the merge which
might mean the community may be unable to profit from the new ABI
until someone volunteers to do it.  Think how long the GC work done on
the branch by Bernd and Richard might have languished had we not
stepped in to integrate it.  This work needs to be done and in
official GCC before IA64 Linux systems go into any kind of serious
use, or there are going to be major headaches down the road -- the
same kind of libc5/libc6 thing will happen all over again.

I think that's perhaps what is still not being understood is:

  o We're not going to break IA32 Linux.

  o We're not going to do stuff willy-nilly without asking the libio
    folks for approval.

  o The only possible breakage will be for -fnew-abi programs,
    which are already not supported -- the new ABI is clearly marked
    as experimental, not for production use, may change, etc.

Branches make sense where there is real risk, or where completion is
uncertain.  For example, the new IA32 back-end branch was an excellent
use of a branch -- had the work been done on the mainline, it probably
would have taken a long time before anything could have been checked
in.  Lots of people would have lost time debugging problems in the new
back-end.  Instead, Richard worked on a branch, interested parties
helped, and finally the result was merged, with minimal impact (except
that our programs run faster now!).  But, the new ABI doesn't affect
anybody -- you only get it if you use -fnew-abi!

What's happenning here is that we're doing open development.

We could keep all our changes locally, give them just to our
customers, prevent the net from helping with the testing and
development load, and prevent people from reaping the benefits of the
new ABI.  Doing them on a branch has many of those same disadvantages.

Up until now we've *never* done any GCC development anywhere but n the
100% open light of day.  We've checked things into GCC when they were
tested and working, and we've fixed problems that arose as quickly as
possible.  We haven't even waited for our customers to pay us for the
work.  We've checked in things as we did the work, allowing everyone
to comment, criticize, test, and improve our efforts.  I'd like to
continue that policy -- I think that's the spirit of open source
development.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    Jeffrey> That is, in effect, what I already stated.  Thus my stand
    Jeffrey> that the libc folks have to be intimately involved in
    Jeffrey> this decision.  It is not a decision that GCC folks
    Jeffrey> should make on their own.

I don't believe I ever suggested making this decision on my own.  I
certainly did not mean to.

This thread *began* with me asking Ulrich for help, after all!

I *of course* intend to run any patches past the libio people.

    >> or (b) is marked very clearly as a local/temporary kludge.

    Jeffrey> This is the kind of stuff that belongs in a local tree,
    Jeffrey> not the GCC sources.

Why?  Without changes, nobody can test the new ABI.  And, there's no
risk to the current ABI.  So, the only thing that go wrong is that we
break the new ABI, but that's clearly not supported by GCC yet.

Are you suggesting I check in the new ABI changes, but *not* the libio
changes, thereby leaving things in a contradictory state?

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@cygnus.com> writes:

    >> From: Mark Mitchell <mark@codesourcery.com> Date: Sun, 19 Dec
    >> 1999 22:26:18 -0800

    >> However, libio has to change to match the new ABI.  That's a
    >> fact.  Obviously, this will be a conditionally compiled change;
    >> the compiler's going to be supporting the old ABI indefinitely.

    Geoff> I think you're misunderstanding the point.  You can't

I don't think so.

    Geoff> conditionally _compile_ anything.  You must make sure
    Geoff> everything will work at runtime, out of the same objects.
    Geoff> This is what 'binary compatibility' means.

Yes.  Let's try this again.

Let's suppose we're running on IRIX.  There's no libio in libc.

  o Now, we can compile with the old C++ ABI or the new C++ ABI.  
    Either is OK -- there's no standard C++ ABI on IRIX.  The user
    might choose this which ABI to use at compiler-configuration time.

    But, the libio in libstdc+++ needs to know about the C++ ABI
    (that's a design choice in libio).  So, we need to make the libio
    in libstdc++ do its by-hand layout of C++ objects in the same 
    way as the C++ compiler.  Which way to do this is known to us
    when we build libio; we've already committed to the C++ ABI to
    use.  Depending on which way the user configured the compiler,
    we'll end up with different versions of libio.
    
  o On Linux, life is more difficult -- libstdc++ and glibc know
    about each other.

    I was thinking that *in principle* one could set up libio
    in libstdc++ analagously to the IRIX situation.  In particular,
    without knowing glibc, just as we do on IRIX.  Of course, one
    would only do this for the new ABI, and only when the system
    libc doesn't already contain the new libio.

    It may be that this is technically infeasible.  It's certainly
    not wrong in principle -- it's in fact the only way to let people
    use the new ABI without upgrading their C library.

To quote Nixon :-), "let me make one thing perfectly clear":

  o I'm not proposing changing *anything* about libio when compiled
    with the existing C++ ABI, which will continue to be supported
    indefinitely.

  o I'm proposing chaning libio, conditionally, when compiled with the
    new C++ ABI.  (This is analagous to multilibbing.)  Without doing
    so, it will not work.

The entirety of a single C++ program must be compiled using -fnew-abi,
or -fno-new-abi -- you can't compile bits and pieces one way.  If you
want to have a library that can be linked with the old ABI and the new
ABI, you compile it twice: once to support the old ABI and once to
support the new one.

That's no different for libio, since it is emulating the C++ ABI, even
though it is written in C.

    >> It would be nice if people can use the new ABI, if only to test
    >> it, on IA32 Linux.  They can't do that unless we can produce a
    >> version of libio that doesn't interact with the C library,
    >> since we can't expect everyone to go upgrading their C library.

    Geoff> Why not?  If the new ABI is stable (or stable enough for
    Geoff> libio), there is no reason not to simply tell people "gcc
    Geoff> version foo requires glibc version bar".

That's not very polite.  That means people with legacy Linux systems
can't use the advances in the compiler.  But, what you say is
possible.

However, we *still* have to change libio for other, non-Linux systems
-- or else nobody can use the new ABI *anywhere*.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Geoff Keating]

> From: Joe Buck <jbuck@synopsys.com>
> Date: Mon, 20 Dec 99 14:42:50 PST
> Cc: jbuck@synopsys.com, mark@codesourcery.com, drepper@cygnus.com,
>         gcc@gcc.gnu.org, libc-alpha@sourceware.cygnus.com
> 
> > My opinion:
> > 
> > The primary purpose of snapshots are for use in the testing and
> > further development of glibc.
> 
> Sorry for the lack of clarity (given that this is going to two lists):
> I was talking about gcc/egcs snapshots, which need to be able to test
> the new C++ API.  We aren't talking about glibc snapshots here.

Did I write that?  I meant, "...development of GCC", I was thinking
about GCC snapshots.  Of course, the rest of my e-mail seems to apply
equally well to both.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991220110443J.mitchell@codesourcery.com >you write:
  > We're not talking about a release here.  We're just talking bout
  > development.
We do not change libio in our development tree without consulting with
and getting agreement from Ulrich and the glibc folks.  That is my
position and I see absolutely no reason to change it.

If you get them to buy into your scheme, then I'll support the code, but
that needs to happen *BEFORE* you install the code into GCC's libio.

  > It isn't really required that libio and glibc change in lockstep.  It
  > certainly doesn't matter on platforms that don't use glibc.
Huh?  If you don't change them in lockstep you introduce incompatibilities
between the version in the linux C library and the version in GCC.  That
has proven to be a majorly stupid thing to do.  We don't want to repeat
that mistake again.  So I'll repeat.  They must change in lock step.


  > It's very hard to move in lock-step; I think there are IA32 Linux
  > users who would like to try out -fnew-abi (to see if it works, to see
  > if it performs better, etc.), but who don't want to upgrade their C
  > libraries.  (I'm one of those users.)  
Yes it is hard; I've been in this situation many times.  Find a way to make
it work.  Find a way that is palatable to the glibc maintainers.

  > Why should we make it hard for people to test the new ABI?
Why should we diverge from glibc?  That just makes our life more difficult.

You've got find another way which doesn't cause our libio to diverge from
the one in glibc.  No ifs ands or buts about it.


jeff

Re: Linux vs. libio

[Joe Buck]

>   In message < 385EAD11.17A3BE8C@cygnus.co.uk >you write:
>   > Or a branch? That would keep it out of the reach of normal users using the
>   > CVS tree, even with -fnew-abi, while still permitting collaboration - which
>   > is presumably the reason Mark wants it checked in to the gcc repository.
> Not a bad idea.  On a branch would be fine by me.  We have used that scheme
> for similar purposes in the past.

Yes, this is the right solution.  People telling Mark to do the work in
his own tree are missing the point that no one can help him test.  But
we want to avoid disrupting others' work.

Re: Linux vs. libio

[Joe Buck]

> My opinion:
> 
> The primary purpose of snapshots are for use in the testing and
> further development of glibc.

Sorry for the lack of clarity (given that this is going to two lists):
I was talking about gcc/egcs snapshots, which need to be able to test
the new C++ API.  We aren't talking about glibc snapshots here.

Re: Linux vs. libio

[Geoff Keating]

> From: Joe Buck <jbuck@synopsys.COM>
> Date: Mon, 20 Dec 99 10:04:02 PST
> 
> Mark Mitchell writes:
> 
> > I assumed that "blanket write privs" applied to everything you get
> > when you do a `cvs co egcs', which includes libio and libstdc++.
> 
> My opinion:
> 
> Blanket write privs are for the purpose of checking things in to CVS.
> Actually doing a release is another matter: for anything that affects
> other projects, the Steering Committee would have to be involved,
> and the SC tries to run by consensus, meaning that if Ulrich Drepper
> were extremely unhappy about something, the SC wouldn't approve its
> release.

My opinion:

The primary purpose of snapshots are for use in the testing and
further development of glibc.  Therefore, it is _not_ a good idea to
break them for no particular reason.  "blanket write privs" only
includes the committing of patches that _work_ (or at least are
believed to work).

> But remember, we never promised that we wouldn't break things in
> *snapshots*.  That's too much of a constraint.

It seems pointless to put stuff in snapshots that will have to be
backed out before a release.

Might I suggest the following flow-chart for Linux GCC ABI changes:

1. Is this patch simply for my internal use only?

   No    Yes------> keep it in your local repository,
   |		    don't put it in CVS at all.  If you think
   |		    others are interested, send it to gcc-patches.
   V
2. Will this patch work on currently shipping Red Hat,
   Debian, Suse, whatever systems, both if you use a new gcc
   on these systems with existing libraries and if you use a new
   gcc to build the libraries but still use old binaries.

   No	 Yes ----->  Then it's probably OK for the mainline tree.
   |
   V
3. Is this patch so important, so useful, that it is acceptable
   to break backwards binary compatibility?

   Yes   No ------>  Then it can't go in, and must be reworked.
   |		     When it's reworked, go back to (2).
   V
4. Can I obtain the consensus of everyone working on or using
   GCC under Linux?

   No    Yes ----->  Budget 1-2 years for building the consensus.  
   |		     Then, once you have that consensus, it can
   |		     go in, at an agreed time, preferably
   |		     with a major version number change for gcc.
   |		     Note that any shared libraries affected
   |		     will also need to bump their version and
   |		     this needs to be coordinated.
   |		     This is a _lot_ of work (budget 50% of
   |		     an engineer/manager's time for the 1-2 years).
   V
5. Your answer at (3) was wrong.
   The patch can't go in, and must be reworked.
   When it's reworked, go back to (2).

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 385EAD11.17A3BE8C@cygnus.co.uk >you write:
  > Or a branch? That would keep it out of the reach of normal users using the
  > CVS tree, even with -fnew-abi, while still permitting collaboration - which
  > is presumably the reason Mark wants it checked in to the gcc repository.
Not a bad idea.  On a branch would be fine by me.  We have used that scheme
for similar purposes in the past.

jeff

Re: Linux vs. libio

[Jonathan Larmour]

Jeffrey A Law wrote:
> 
>   >  or (b) is marked
>   > very clearly as a local/temporary kludge.
> This is the kind of stuff that belongs in a local tree, not the GCC sources.

Or a branch? That would keep it out of the reach of normal users using the
CVS tree, even with -fnew-abi, while still permitting collaboration - which
is presumably the reason Mark wants it checked in to the gcc repository.

It's a bit more effort to merge the branch back in, but it can be done
discretely, without breaking anything.

Jifl

Re: Linux vs. libio

[Jeffrey A Law]

  In message < m24sddz7ak.fsf@magnus.bothner.com >you write:
  > I haven't suggested that.  What I said is that no changes should be
  > checked into areas of the gcc cvs tree that we don't own (such as
  > texinfo and any libc parts of libio) unless either (a) it has first
  > been checked into the "upstream" master source tree,
That is, in effect, what I already stated.  Thus my stand that the libc
folks have to be intimately involved in this decision.  It is not a
decision that GCC folks should make on their own.

  >  or (b) is marked
  > very clearly as a local/temporary kludge.
This is the kind of stuff that belongs in a local tree, not the GCC sources.


[ ... ]

  > This is similar to the policy Cygnus has (had) for gcc: Nothing should
  > be in the Cygnus internal cvs tree unless it is also in the external
  > tree, or marked with special markers in the code.
Correct.  Note that Cygnus doesn't tend to push those local/temporary 
kludges into the official GCC sources, nor should anyone else.  Those
kludges are kept as local changes.


jeff

Re: Linux vs. libio

[Per Bothner]

Kevin Atkinson <kevinatk@home.com> writes:

> Maybe I am missing something but couldn't the io code in linux be
> written in pure C

I believe it is.

> with a few extensions to allow C++ wrapper classes to
> effectively use the io services.   Like say provide inline functions (or
> macros) to manipulate the buffer efficiently.

It is more complicated than that.  The point is *extensibility*
through jump tables (in C) or vtables (in C++).  The standard
C++ libary is a set of classes (and templates) that use
inheritance and virtual functions.

For exmaple, the C++ version of `getc' has an inline path that
is fast and similar to C:
        if (str->read_pointer < str->read_limit)
          return *str->read_pointer++;
        else
          return handle_slow_case(str);

So what we in GNU is use the *same* inline code for both C and C++.
I.e. the C++ methods and the C getc access the *same* read_pointer.

For C++, handle_slow_case(str) actually does a virtual function
call to a method which depends on the run-time type of (*str)
(which is not necessarily known at compile-time).  For example,
we may be reading from a file, a string, a socket, a wrapper
around readline, or an Emacs-style buffer.  In standard C,
there is only only kind of FILE, so we can call the appropriate
function directly.  However, glibc has long provided similar
extensible I/O as C++, but using jump tables of function pointers.
It makes sense to use the same format for those jump tables as
the C++ virtual function tables, hence some of the ugly macros
in the libio source.

One can write a C++ wrapper class (say stdiobuf) that implements the
C++ functionality, but uses the C stdio buffers.  The problem is that
this is slow, since you have to call handle_slow_case on every
character.  You can force that by setting read_pointer == read_limit
(for those streams that are aliased to stdio).  Alternatively, you can
make sure to use the same value of read_pointer for C FILEs and C++
filebufs.  However, then either you get extra overhead (indirection)
or you have to merge the data structures for C and C++.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> This may be an example of such a pinch.  The new ABI has to be tested.
> Requiring all testers to rebuild their glibc means that there will be
> a lot less testing, meaning a lower-quality release.

I haven't suggested that.  What I said is that no changes should be
checked into areas of the gcc cvs tree that we don't own (such as
texinfo and any libc parts of libio) unless either (a) it has first
been checked into the "upstream" master source tree, or (b) is marked
very clearly as a local/temporary kludge.  The former is much to be
preferred.  I certainly don't want to require people to re-build
glibc, unless they have a glibc *snapshot* and *want to* re-build
everything.

This is similar to the policy Cygnus has (had) for gcc: Nothing should
be in the Cygnus internal cvs tree unless it is also in the external
tree, or marked with special markers in the code.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Joe Buck]

I wrote:

> > Please note that I'm only one SC member.  If other members object to
> > what I'm saying above, we'd have to resolve it.

Per Bothner (another SC member) writes:

> I do object, or rather:  we should be very careful.  I think checking
> in changes for code that we do not "own" can very easily lead to
> divergence.

It's not as much of a difference as it might appear, since I'd want
things re-sync'd before releases, and I expect everyone with blanket
write privileges to "be very careful".

> In a pinch we can put in a temporary kludge, but only if it is
> clearly marked as such.

This may be an example of such a pinch.  The new ABI has to be tested.
Requiring all testers to rebuild their glibc means that there will be
a lot less testing, meaning a lower-quality release.

Re: Linux vs. libio

[Kevin Atkinson]

Per Bothner wrote:

> Duh.  libio defines some *C++* classes, whose specification
> was (partially) specified by existing (de facto) standards.
> Using plain C misses the point.  Libio was written to facilitate
> integration of C and C++ I/O, which is *required* by the C++ standard.
> 
> There is (as always) a design tradeoff.  As I see it,
> unified C/C++ I/O gives us:
> + Improved performance.
> + Conceptual elegance of uniting the I/O facilities of C/C++.
> + Easier mixing of C and C++ code (a GNU extension).
> + Sharing some tricky code (e.g. buffer management).
> 
> On the other hand, it costs us:
> - Extra "hair".
> - Compatibility issues made more complicated when two ABIs
>   depend so closely on each other.

Maybe I am missing something but couldn't the io code in linux be
written in pure C with a few extensions to allow C++ wrapper classes to
effectively use the io services.  Like say provide inline functions (or
macros) to manipulate the buffer efficiently.

-- 
Kevin Atkinson
kevinatk@home.com
http://metalab.unc.edu/kevina/

Re: Linux vs. libio

[Geoff Keating]

> From: Mark Mitchell <mark@codesourcery.com>
> Date: Sun, 19 Dec 1999 22:26:18 -0800

> However, libio has to change to match the new ABI.  That's a fact.
> Obviously, this will be a conditionally compiled change; the
> compiler's going to be supporting the old ABI indefinitely.  

I think you're misunderstanding the point.  You can't conditionally
_compile_ anything.  You must make sure everything will work at
runtime, out of the same objects.  This is what 'binary compatibility'
means.

> It would be nice if people can use the new ABI, if only to test it, on
> IA32 Linux.  They can't do that unless we can produce a version of
> libio that doesn't interact with the C library, since we can't expect
> everyone to go upgrading their C library.

Why not?  If the new ABI is stable (or stable enough for libio), there
is no reason not to simply tell people "gcc version foo requires glibc
version bar".

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Per Bothner]

Joe Buck <jbuck@synopsys.COM> writes:

> Please note that I'm only one SC member.  If other members object to
> what I'm saying above, we'd have to resolve it.

I do object, or rather:  we should be very careful.  I think checking
in changes for code that we do not "own" can very easily lead to
divergence.  It is all too easy to put in a fix in our sources, and
forget about it.  Unless it also gets into the master sources, we now
have a potential future maintainance/merge problem.

As a general rule, we should only check in changes to foreign
code if it has been approved by the "upstream" maintainer.
Ideally, we should only make changes that will appear in some
future official release of that code, preferably the next one!
In a pinch we can put in a temporary kludge, but only if it is
clearly marked as such.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Joe Buck]

>     Joe> I think that the glibc folks should
>     Joe> approve patches in the libraries with ABI-visible effects
>     Joe> before they go to release branches, but we shouldn't need to
>     Joe> wait for such approval to check something into the CVS
>     Joe> archive.  After all, some problems won't be detected until
>     Joe> people test snapshots anyway.

Mark Mitchell writes:

> Thanks for clarifying!

Please note that I'm only one SC member.  If other members object to
what I'm saying above, we'd have to resolve it.

Re: Linux vs. libio

[Per Bothner]

Joern Rennecke <amylaar@cygnus.co.uk> writes:

> C++ was known to be volatile when libio was written.
> The compatibility issues we have now could have been avoided if the API
> of libio had been designed to use plain C.

Duh.  libio defines some *C++* classes, whose specification
was (partially) specified by existing (de facto) standards.
Using plain C misses the point.  Libio was written to facilitate
integration of C and C++ I/O, which is *required* by the C++ standard.

There is (as always) a design tradeoff.  As I see it,
unified C/C++ I/O gives us:
+ Improved performance.
+ Conceptual elegance of uniting the I/O facilities of C/C++.
+ Easier mixing of C and C++ code (a GNU extension).
+ Sharing some tricky code (e.g. buffer management).

On the other hand, it costs us:
- Extra "hair".
- Compatibility issues made more complicated when two ABIs
  depend so closely on each other.

One can rationally argue for or against the unification.
An argument can be made that unification is "doing the right
thing" and well within the Gnu tradition of sacrificing
implementation complication for better performance or functionality.
What gets really annoying when C programmers make judgemental
statements without understanding the issues.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    >> The problem is that part of the libio API is the C++ vtables,
    >> so any ABI change that modifies the layout of the vtable (such
    >> as -fvtable-thunks) or how the vtable is found breaks libio.

    Jeffrey> Right.  This is precisely why I stated that we should not
    Jeffrey> change libio without changing glibc too.  They need to
    Jeffrey> change in lock-step.

We're not talking about a release here.  We're just talking bout
development.

It isn't really required that libio and glibc change in lockstep.  It
certainly doesn't matter on platforms that don't use glibc.

What's required is that if libio changes on IA32 Linux (which, under
my proposal, it would only do when compiled with -fnew-abi), then it
needs to be simultaneously disentangled from glibc, and work in the
way that libio works with other C libraries.  If that disentanglement
can't be done, then it can't be done -- people will only be able to
use -fnew-abi on non IA32 Linux platforms.

If we decide to *release* an IA32 Linux compiler where -fnew-abi is a
supported option (which we have not yet done to date), then we
probably want to do so only when there's a C library that has the new
libio.  That's not a requirement for releasing an IRIX compiler where
-fnew-abi is a supported option.

It's very hard to move in lock-step; I think there are IA32 Linux
users who would like to try out -fnew-abi (to see if it works, to see
if it performs better, etc.), but who don't want to upgrade their C
libraries.  (I'm one of those users.)  

Why should we make it hard for people to test the new ABI?

There are really only a few options that I can think of:

  1. Don't implement -fnew-abi.

  2. Implement -fnew-abi, but ignore the fact that libio is broken
     for the new ABI.  

     This solution makes -fnew-abi difficult to test, and makes
     an -fnew-abi compiler useless to the community, unless they 
     get iostreams from somewhere else.  (For example, SGI has
     a conforming implementation available at
     http://www.sgi.com/Technology/STL .)

  3. Implement the libio changes in the glibc version.  That will 
     make it considerably more difficult to test the work we are
     really trying to do: implement the new ABI.  Just mucking with
     the build process in GCC to make this work will be a
     nuisance.  Then, we have to either wait until the libio in glibc
     is considered stable and move the result back into the compiler,
     or backport the changes to the version in the compiler.

  4. Implement the libio changes in the compiler version.  That
     makes testing easy, and doesn't risk changing the libio to 
     a newer, less stable version.  Then, forward-port the changes
     from there into glibc.

Option 4 makes sense.  It requires minimal changes.  It allows us to
implement and experiment with -fnew-abi most effectively.  Once things
are working smoothly, we can move the changes into glibc.  We'll
commit to helping with that process.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joern Rennecke]

> Jeffrey A Law <law@cygnus.com> writes:
> 
> > I don't really see the need to debate this -- it's a fact of life that we
> > have to deal with this braindamage.

Ulrich Drepper:
> You don't want any discussion but still insist on insulting other
> people.  Very strange.  Plus that you keep saying this does mean you
> haven't spend a second thinking about it.

C++ was known to be volatile when libio was written.
The compatibility issues we have now could have been avoided if the API
of libio had been designed to use plain C.

Well, now that the damage is done, how about introducing an attribute
for functions and structures that the old ABI is to be used.
The header files that declare libio functions and structures for other
code to use could bear this attribites, so that we can freeze the libio
API while the default ABI changes.

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Joe" == Joe Buck <jbuck@synopsys.COM> writes:

    Joe> things for users.  I think that the glibc folks should
    Joe> approve patches in the libraries with ABI-visible effects
    Joe> before they go to release branches, but we shouldn't need to
    Joe> wait for such approval to check something into the CVS
    Joe> archive.  After all, some problems won't be detected until
    Joe> people test snapshots anyway.

Thanks for clarifying!

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Joe Buck]

Mark Mitchell writes:

> I assumed that "blanket write privs" applied to everything you get
> when you do a `cvs co egcs', which includes libio and libstdc++.

My opinion:

Blanket write privs are for the purpose of checking things in to CVS.
Actually doing a release is another matter: for anything that affects
other projects, the Steering Committee would have to be involved,
and the SC tries to run by consensus, meaning that if Ulrich Drepper
were extremely unhappy about something, the SC wouldn't approve its
release.

But remember, we never promised that we wouldn't break things in
*snapshots*.  That's too much of a constraint.

> But, there's still a question remaining: are people with blanket write
> privs allowed to change libio/libstdc++ without your approval, or not?
> And, for that matter, it appears that Gabriel Dos Reis is also listed
> as a maintainer of "c++ runtime libs" as well, so may I assume that
> even if approval is required, Gaby's approval is good enough?

I think we need a compromise between measures that would slow down
development too much, and measures that would break things for users.  I
think that the glibc folks should approve patches in the libraries with
ABI-visible effects before they go to release branches, but we shouldn't
need to wait for such approval to check something into the CVS archive.
After all, some problems won't be detected until people test snapshots
anyway.

Re: Linux vs. libio

[Per Bothner]

Jamie Lokier <jamie.lokier@cern.ch> writes:

> The down side is it's not at all portable, so code that's written on
> Linux will output differently on other platforms.

Firstly, is not a downside if glibc provides well-defined expected
behavior where other platforms don't.  Secondly, the C++ standard
*does* require the standard streams cin/cout/cerr to be syncronized
with stdin/stdout/stderr.  You can do this portably, but you do an
extra useless plus a function call on every character.

> Only the /very/ few fields that are used by inline input/output
> functions need to be shared. Since they are simple data fields, sharing
> those would be very easy even with many different kinds of C++ ABI.
> 
> Everything else, including vtable format and placement does not need to
> be shared.  Just have the C library call functions (not virtual ones)
> like "flush_my_output_stream" etc., and those functions would contain
> the vtable lookup code.

I believe we do that.  At least we used to.  The intent was to partly
insulate user code from ABI changes.  However, you still need to
co-ordinate the C and C++ data structures.  If you do that, it seems
logical to *share* the data structures.  Otherwise, you have to use
extra indirection.  (You have to share the pointer(s) for the current
input/output position:  The same actual pointers have to be logically
in both the FILE and the streambuf.  So either one of these data
structures has to use an extra indirction, or they have to be the
same object, or at least the objects have to overlap.)
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < u9g0wxbk5g.fsf@yorick.cygnus.com >you write:
  > >>>>> Jeffrey A Law <law@cygnus.com> writes:
  > 
  >  > You will need to work with the glibc folks to find a solution that makes
  >  > sense to them as we do not want to start changing the API of the libio
  >  > contained in GCC without making corresponding changes to the libio
  >  > contained in glibc.
  > 
  > The problem is that part of the libio API is the C++ vtables, so any ABI
  > change that modifies the layout of the vtable (such as -fvtable-thunks) or
  > how the vtable is found breaks libio. 
Right.  This is precisely why I stated that we should not change libio without
changing glibc too.  They need to change in lock-step.

That means that the libc folks have to be intimately involved in the decisions
we make regarding the ABI/API of libio (and IMHO have the ultimate yea/nay
decision for such changes).

  > This has caused us a lot of trouble over the years; perhaps the answer is
  > to change that design,
Possibly, but that does not help us with the existing glibc installations.

  > but this *is* supposed to be the all-singing, all-dancing, ultimate C++
  > ABI.
:-)  I'll be floating on air when it's all ready.  I really want to put
the ABI issues behind us once and for all.



jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jamie" == Jamie Lokier <jamie.lokier@cern.ch> writes:

    Jamie> <evil> You might be able to do it by reserving a field in
    Jamie> the new structures at the right place to contain the
    Jamie> old-Glibc-compatible vtable pointer, and having two

Interesting idea.  I think this might have been what Ulrich was
suggesting as well in one of his messages, but it didn't make sense to
me at the time.  Something like this might indeed help solve some of
the problems.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "H" == H J Lu <hjl@lucon.org> writes:

    H> Take a look at how libc5 is handled in libio in gcc. libio in
    H> gcc 2.95.2 is quite different from libio in libc5.

Thanks for your suggestion.  I'll see if that helps.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jason Merrill]

>>>>> Jeffrey A Law <law@cygnus.com> writes:

 > You will need to work with the glibc folks to find a solution that makes
 > sense to them as we do not want to start changing the API of the libio
 > contained in GCC without making corresponding changes to the libio
 > contained in glibc.

The problem is that part of the libio API is the C++ vtables, so any ABI
change that modifies the layout of the vtable (such as -fvtable-thunks) or
how the vtable is found breaks libio.  This has caused us a lot of trouble
over the years; perhaps the answer is to change that design, but this *is*
supposed to be the all-singing, all-dancing, ultimate C++ ABI.

Jason

Re: Linux vs. libio

[Ulrich Drepper]

Jeffrey A Law <law@cygnus.com> writes:

> I don't really see the need to debate this -- it's a fact of life that we
> have to deal with this braindamage.

You don't want any discussion but still insist on insulting other
people.  Very strange.  Plus that you keep saying this does mean you
haven't spend a second thinking about it.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Jamie Lokier]

Per Bothner wrote:
> Jeffrey A Law <law@cygnus.com> writes:
> 
> > While I believe it was an unbelievably dumb thing for the glibc folks to
> > tie libio into the C library,
> 
> In other words, you don't think it is a reasonable goal to allow C
> stdio FILEs to be shared with C++ iostream streambufs?

The down side is it's not at all portable, so code that's written on
Linux will output differently on other platforms.

> The best way to synchronize (say) stdout with cout seems to be for
> them both to use the same buffer.  The requires (at least if you want
> efficiency) for them to use compatible data structures,

Only the /very/ few fields that are used by inline input/output
functions need to be shared.  Since they are simple data fields, sharing
those would be very easy even with many different kinds of C++ ABI.

Everything else, including vtable format and placement does not need to
be shared.  Just have the C library call functions (not virtual ones)
like "flush_my_output_stream" etc., and those functions would contain
the vtable lookup code.  Or alternatively, have an explicit /C/ vtable
as one of the shared fields and keep it that way for the C++ version.
C++ streams don't have to call that vtable; they merely have to provide
it.

-- Jamie

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Jeffrey" == Jeffrey A Law <law@cygnus.com> writes:

    >> To get it working at all, he must change libio, and he wants to
    >> avoid rebuilding his Linux distribution to do so. I cannot find
    >> anything unwise here.

    Jeffrey> He's more than welcome to do this on his local box.  But
    Jeffrey> it's not something that belongs in our source tree.

Yes, it does.  There are three distinct things happenning here:

  o Implementation of -fnew-abi.

    We all agree this is a Good Thing.  Which platforms use it by
    default, when they do so, etc., is not something I've got any
    feelings about.  But, we want the new ABI to exist, and it will
    be the default ABI on IA64, I assume, since that's the ABI
    specified there.

    So, this goes in the tree.

  o Adjustments to libio.

    The current libio *will not work* without modification with the
    new ABI, even on non-Linux systems.  Suppose you're running 
    IRIX, and suppose you want to use the new ABI.  Or suppose
    you're running IA64 and you have to use the new ABI.  You need
    a modified libio.
    
    Of course, we need to conditionally compile the changes so that
    old ABI systems (like RH Linux) still work.  But, we can't
    support the new ABI without these changes, so they need to 
    go in the tree.

  o Configury changes to that building with -fnew-abi, on a Linux
    sytem using a glibc that contains a libio built with the old
    ABI, leaves you with a working compiler.

    If someone configures their tree using -fnew-abi on, say,
    RH 6.x, they'll get a totally non-working libio unless we
    turn on the new ABI in libio.  If that conflicts with the C
    library, and we don't sort out that interaction, they'll be hosed.

    There's no reason to say that people on IA32 Linux can't use
    the new ABI.  Just because libio happens to be in libc shouldn't
    (abstractly) sentence them to a worse fate than people running
    IRIX (where libio is not in libc).  The libio library knows how
    to deal with a "foreign" libc, and it should be possible to treat
    glibc as "foreign".  It may or may not be technically feasible
    to achieve this.

    I would argue that if it is achievable, this hunk should go in the
    tree as well.

I know there may be adverse consequences on performance as a result,
and I fully agree that the desirable configuration is to have a libc
and libstdc++ that share a libio.  But, for non-glibc systems we need
a solution, and for IA32 Linux I don't see why we shouldn't support
-fnew-abi configurations, if that's possible.

Believe it or not, I am not a *complete* idiot.  I'm not going to
somehow go breaking every Linux system on the planet, or something.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Jeffrey A Law]

  In message < m21z8h1rmy.fsf@magnus.bothner.com >you write:
  > Jeffrey A Law <law@cygnus.com> writes:
  > 
  > > While I believe it was an unbelievably dumb thing for the glibc folks to
  > > tie libio into the C library,
  > 
  > In other words, you don't think it is a reasonable goal to allow C
  > stdio FILEs to be shared with C++ iostream streambufs? 
  > The best way
  > to synchronize (say) stdout with cout seems to be for them both to use
  > the same buffer.  The requires (at least if you want efficiency) for
  > them to use compatible data structures, and thus libio has to be tied
  > to the C library.  (Note: This is not new to glibc; it has been that
  > way in Linux for many years.)
I don't really see the need to debate this -- it's a fact of life that we
have to deal with this braindamage.  So, let's find a way to deal with it
rather than debating the decision to integrate libio with glibc here.

jeff

Re: Linux vs. libio

[Jamie Lokier]

Mark Mitchell wrote:
> With the new ABI, pointers to virtual tables come at a different point
> in the objects.  Bases are laid out in different orders.  There are no
> longer pointers to virtual bases.  Some of that doesn't matter; some
> of it does.  Given:
> 
>   struct _ios_fields { ... };
>   class ios : public _ios_fields { ... };
> 
> it is no longer the case that you can cast an `ios *' to an
> `_ios_fields *' without having to do modify the pointer value;

> More importantly, the same holds for streambuf and _IO_FILE.  You
> can't cast an _IO_FILE to a streambuf, or vice versa, without
> modifying the pointer value.

<evil>
You might be able to do it by reserving a field in the new structures at
the right place to contain the old-Glibc-compatible vtable pointer, and
having two separate vtables.  Only one vtable (and corresponding
structure layout) is active at a time.  The other points to a set of
thunks.  When you call a thunk, it modifies the vtable pointers to swap
which is the active vtable, reorders the data fields for the other ABI
and continues from there.  
</evil>

;-)

-- Jamie

Re: Linux vs. libio

[Per Bothner]

Jeffrey A Law <law@cygnus.com> writes:

> While I believe it was an unbelievably dumb thing for the glibc folks to
> tie libio into the C library,

In other words, you don't think it is a reasonable goal to allow C
stdio FILEs to be shared with C++ iostream streambufs?  The best way
to synchronize (say) stdout with cout seems to be for them both to use
the same buffer.  The requires (at least if you want efficiency) for
them to use compatible data structures, and thus libio has to be tied
to the C library.  (Note: This is not new to glibc; it has been that
way in Linux for many years.)
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 199912201435.PAA01388@loewis.home.cs.tu-berlin.de >you write:
  > It seems there is some confusion about the scope of what Mark attempts
  > to do.
  > 
  > He is working, in private, on implementing the new C++ ABI. Because he
  > is lacking an IA64 machine, he uses his private ia32 machine to do so.
I understand this.

  > To get it working at all, he must change libio, and he wants to avoid
  > rebuilding his Linux distribution to do so. I cannot find anything
  > unwise here.
He's more than welcome to do this on his local box.  But it's not something
that belongs in our source tree.

  > Please, can we leave the decision what to work on to the individual
  > developer, and only judge the quality of the work when we see actual
  > patches?
It's not an issue of quality, but of what belongs in our source tree.  I
still see no reason for this code to go in our source tree.

jeff

Re: Linux vs. libio

[Martin v. Loewis]

> I think this is rather unwise, and ultimately the wrong approach to solving
> whatever problem you're working on.

It seems there is some confusion about the scope of what Mark attempts
to do.

He is working, in private, on implementing the new C++ ABI. Because he
is lacking an IA64 machine, he uses his private ia32 machine to do so.
To get it working at all, he must change libio, and he wants to avoid
rebuilding his Linux distribution to do so. I cannot find anything
unwise here.

Now he asks libio experts on how he should implement these changes, on
a technical level. All he gets as a response is "no, you don't want to
do that" (except for HJ Lu, who was giving some actual advice).

Please, can we leave the decision what to work on to the individual
developer, and only judge the quality of the work when we see actual
patches?

Having an independent copy of libio is only a stop-gap. We can look
for ways to integrate both copies into a single one later. That may or
may not be feasible. IMHO, changing the class layout is still the
right thing to do, and libio versioning should not stop us.

Regards,
Martin

Re: Linux vs. libio

[Jeffrey A Law]

  In message < 19991219183255Q.mitchell@codesourcery.com >you write:
  >   In other words, what I'm looking to do is to have two distinct,
  > disentangled, copies of libio: one in libc, to handle stdio, and one
  > in libstdc++ to handle iostreams.  How can this be done?
I think this is rather unwise, and ultimately the wrong approach to solving
whatever problem you're working on.

While I believe it was an unbelievably dumb thing for the glibc folks to
tie libio into the C library, it's done and the GCC folks have to deal with
it.

You will need to work with the glibc folks to find a solution that makes sense
to them as we do not want to start changing the API of the libio contained
in GCC without making corresponding changes to the libio contained in glibc.


jeff

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> Fortunately, the changes are relatively minimal: almost no code
    >> -- just a few changes in structure definitions and in a few
    >> macros.

    Ulrich> Adding these changes will probably be ok for other
    Ulrich> platforms (since there is no backward compatibility to be
    Ulrich> maintained) and it is also ok *for test purposes* to use
    Ulrich> this changed code on Linux, but no production release
    Ulrich> should be made with these changes and people should be
    Ulrich> discouraged to distribute libstdc++ versions compiled in
    Ulrich> this way.

Now I feel like you've heard what I was saying.

That's *exactly* what the purpose of the changes is -- other platforms
and testing.  That's what I said all along.

I can also imagine that we might have a configuration using the
un-integrated versino for people who have "old" libcs (like RH 6.x)
but want to use -fnew-abi.

Obviously, we're not going to ship anything with -fnew-abi until it's
stable, tested, etc.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> Fortunately, the changes are relatively minimal: almost no code --
> just a few changes in structure definitions and in a few macros.

This is not the point.  As Geoff pointed out the situation on Linux at
least is therefore wrt C++ I/O pretty good because of the tight
integration of the C and C++ functions.  Besides this is much more
light-weight and in future (after the merge with my development
version) it'll allow full wide-char and locale integration, something
which cannot 100% be achieved on other platforms.  This is not
possible if you now simply change the API and actually generate libio
code in libstdc++ which does not use the code in the libc.

Adding these changes will probably be ok for other platforms (since
there is no backward compatibility to be maintained) and it is also ok
*for test purposes* to use this changed code on Linux, but no
production release should be made with these changes and people should
be discouraged to distribute libstdc++ versions compiled in this way.
libstc++ is for various reasons generated most often as a shared
object and therefore it is really A Bad Thing(tm) to change the API in
an incompatible way just to change it again a bit later.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Mark" == Mark Mitchell <mark@codesourcery.com> writes:

    Mark> Fortunately, the changes are relatively minimal: almost no
    Mark> code -- just a few changes in structure definitions and in a
    Mark> few macros.

Actually, I lied. :-)  That's the conceptual picture -- there are
actually relatively many places where the code will get changed -- but
in a reasonably uniform way.

For example, we currently use the same set of macros (like _IO_XSPUTN)
on both `streambuf *'s and `_IO_FILE *'s.  That doesn't work any more
since the pointer types are not interchangeable.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> Still, neither you nor I can answer my procedural question. :-)
    >> It's a steering committe issue: they (poor things!), not us,
    >> get to decide these questions. :-)

    Ulrich> Not all code in egcs is governed by the steering
    Ulrich> committee.  Just think about texinfo.  Simply because code
    Ulrich> was added to egcs for convenience reasons does not make
    Ulrich> the SC the owner.  It can decide about the future of that
    Ulrich> copy (= branch) but we all certainly want to avoid
    Ulrich> branches.

The FSF is the "owner" of all of this software.  :-)

But, I know what you mean.

It's true that the master version of texinfo is not in GCC.  It's also
true that I would, if for some reason the need arose, feel it was
reasonable for me to modify the version of texinfo in GCC, without
contacting the texinfo maintainers first.  Of course, if the changes
were generally useful, we'd submit them back to the texinfo folks as
well.

My understanding is that the SC *does* get to decide what to do about
stuff in the GCC tree.  That doesn't mean it has anything to say about
the *master* versions of those packages that are officially maintained
elsewhere.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> If you want to update the version of libio in the compiler, by
    >> all means go ahead!

    Ulrich> I would have done this if it would make sense.  But it
    Ulrich> does not since the changes are not yet fully tested and
    Ulrich> probably not even complete.  Adding all the wide char
    Ulrich> stuff is non-trivial.

That's what I figured.  So, we're between the proverbial devil and the
deep blue sea: we can't easily test -fnew-abi without making some
changes in the version used with the compiler, and we can't easily
upgrade that version.  So, we'll have to make the changes, and merge
later, I think.

Fortunately, the changes are relatively minimal: almost no code --
just a few changes in structure definitions and in a few macros.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> Still, neither you nor I can answer my procedural question. :-) It's a
> steering committe issue: they (poor things!), not us, get to decide
> these questions. :-)

Not all code in egcs is governed by the steering committee.  Just
think about texinfo.  Simply because code was added to egcs for
convenience reasons does not make the SC the owner.  It can decide
about the future of that copy (= branch) but we all certainly want to
avoid branches.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> If you want to update the version of libio in the compiler, by all
> means go ahead!

I would have done this if it would make sense.  But it does not since
the changes are not yet fully tested and probably not even complete.
Adding all the wide char stuff is non-trivial.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Mark Mitchell <mark@codesourcery.com> writes:

    >> But, there's still a question remaining: are people with
    >> blanket write privs allowed to change libio/libstdc++ without
    >> your approval, or not?

    Ulrich> I certainly reserve all rights for the libio parts as I'm
    Ulrich> probably the only one who sees the whole picture (libio in
    Ulrich> libstdc++ and in glibc).  As for the rest of libstdc++,

I'm not going to tread on your turf.  Obviously, you feel very
strongly about this.  And, obviously, I'd consult you anyhow -- I
already did!

Still, neither you nor I can answer my procedural question. :-) It's a
steering committe issue: they (poor things!), not us, get to decide
these questions. :-)

    Ulrich> that there were enough tests in the first place).  Plus:
    Ulrich> the version in libstdc++ is not the latest one (i.e., not
    Ulrich> the development version).

But, it is the version we ship with the compiler.  So, it's the only
one we can easily update to work with the new ABI, at this point.
Obviously, at some point, we'll have to merge the changes -- that
happens anytime a subproject is shared between two others.

If you want to update the version of libio in the compiler, by all
means go ahead!

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> This is still no reason to break compatibility.  If the
    Ulrich> virtual tables require a new layout one can add new ones.

OK, I'm listening.  I don't understand what you mean, but if you can
show me how to do this, and avoid having to have two separate libios
(one in libstdc++, and one in libc), I'll be all too happy to
implement it.

With the new ABI, pointers to virtual tables come at a different point
in the objects.  Bases are laid out in different orders.  There are no
longer pointers to virtual bases.  Some of that doesn't matter; some
of it does.  Given:

  struct _ios_fields { ... };
  class ios : public _ios_fields { ... };

it is no longer the case that you can cast an `ios *' to an
`_ios_fields *' without having to do modify the pointer value; the
vtable pointer comes before the _ios_fields object in an ios.

More importantly, the same holds for streambuf and _IO_FILE.  You
can't cast an _IO_FILE to a streambuf, or vice versa, without
modifying the pointer value.

That means the (old) macros (like JUMP1) that take an _IO_FILE *, go
find the vtable, and jump through it, don't work when given an
_IO_FILE * that came from a (new) streambuf; the vtable is now
*before* the _IO_FILE, not after.  Since there is code that uses those
macros already compiled into libc, I don't see how to make them
interoperate.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> But, there's still a question remaining: are people with blanket write
> privs allowed to change libio/libstdc++ without your approval, or not?

I certainly reserve all rights for the libio parts as I'm probably the
only one who sees the whole picture (libio in libstdc++ and in glibc).
As for the rest of libstdc++, it's not me.  Rather Benjamin and
Gabriel, so they have to speak for themselves.  And yes, I think the
write access you got from Jeff and the gcc people does not apply to
the libraries.  Esp libio has far too many implications then you can
test for by simply running libstdc++ tests (not that there were enough
tests in the first place).  Plus: the version in libstdc++ is not the
latest one (i.e., not the development version).

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich> Wait a minute, why do you think you can change libio?  If
    Ulrich> something has to be behave different there must be a
    Ulrich> different solution found.  You are not with the compiler,

We need a procedural clarification here.

I assumed that "blanket write privs" applied to everything you get
when you do a `cvs co egcs', which includes libio and libstdc++.

Obviously, making changes there, as with any part of the system,
requires care, and entails risk.  Just as I would not go willy-nilly
hacking around in the ARM back-end (which I know next to nothing
about), I would not go willy-nilly hacking about thinks in libio.
And, were I to change the ARM back-end, and were Richard Earnshaw to
tell me I had done so incorrectly, I would certainly defer to his
judgement.

But, there's still a question remaining: are people with blanket write
privs allowed to change libio/libstdc++ without your approval, or not?
And, for that matter, it appears that Gabriel Dos Reis is also listed
as a maintainer of "c++ runtime libs" as well, so may I assume that
even if approval is required, Gaby's approval is good enough?

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

> The libio tricks whereby there are C versions of C++ objects (complete
> with vtables, etc.) will still work, but the objects are going to have
> to be laid out differently, because the C++ objects are being laid out
> differently.

This is still no reason to break compatibility.  If the virtual tables
require a new layout one can add new ones.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Geoff" == Geoff Keating <geoffk@envy.cygnus.com> writes:

    Geoff> It may be that you will have to change glibc to be
    Geoff> compatible with the new layout.  You've already changed
    Geoff> glibc to be compatible with the new EH so you will be
    Geoff> requiring a new version of glibc anyway.

I think you mean the new atexit stuff.  Technically, that can go in
libstdc++, if necessary, and just replace the bits in libc.

But, you are correct -- in the end there will need to be a new libc.
It may or may not be deployed on IA32 Linux; that's not something I'll
decide.

However, I'm sure it will be deployed on IA64 Linux.  It's also likely
to be used on existing non-Linux platforms, like Solaris or IRIX.

Right now, I'm just trying to test the work, and the systems I have
handy are IA32 Linux.

    Geoff> You don't want to do that.  It would cause C stdio and C++
    Geoff> iostreams to be buffered separately causing output to be
    Geoff> interleaved wrongly.  It would also break the nice
    Geoff> stdio/iostreams interface that took so much work to get
    Geoff> right.

All these things are no doubt true.

I know why there's interdependency between these things, and I know we
want to keep that in the future.

However, libio has to change to match the new ABI.  That's a fact.
Obviously, this will be a conditionally compiled change; the
compiler's going to be supporting the old ABI indefinitely.  

It would be nice if people can use the new ABI, if only to test it, on
IA32 Linux.  They can't do that unless we can produce a version of
libio that doesn't interact with the C library, since we can't expect
everyone to go upgrading their C library.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Mark Mitchell]

>>>>> "Ulrich\" == Ulrich Drepper <drepper@cygnus.com> writes:

    Ulrich\> Wait a minute, why do you think you can change libio?  If

Because it's under the GPL? :-) :-) :-)

Please don't get territorial.  I'm not about to go releasing some
weird forked version of anything.  I asked you for help because I know
you're the expert.

I'm just doing work.  You can always choose to accept or reject any
changes in the end; that's your privilege as libio maintainer.

    Ulrich\> something has to be behave different there must be a
    Ulrich\> different solution found.  You are not with the compiler,

There *is* going to be a new C++ ABI.  We're all agreed on that, and
we're agreed on what it will be.  It's not some whim of mine; it's an
industry-standard cross-OS ABI.

The libio tricks whereby there are C versions of C++ objects (complete
with vtables, etc.) will still work, but the objects are going to have
to be laid out differently, because the C++ objects are being laid out
differently.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: Linux vs. libio

[Geoff Keating]

Mark Mitchell <mark@codesourcery.com> writes:

> Folks --
> 
>   Does anyone know how, on Linux, to build the libio (in CVS with GCC)
> in such a way as to be entirely disentangled from the C library?
> (Except in the ways that libio would normally be entangled with the C
> library on any non-glibc system.)
> 
>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.

I think this is a sign that you cannot change libio like that.  These
routines are interfaces between glibc and libio.  Yes, I know, it's
evil, but it's no worse than libgcc which is the same (ever heard of
__register_frame_pointer?).

It may be that you will have to change glibc to be compatible with the
new layout.  You've already changed glibc to be compatible with the
new EH so you will be requiring a new version of glibc anyway.

>   In other words, what I'm looking to do is to have two distinct,
> disentangled, copies of libio: one in libc, to handle stdio, and one
> in libstdc++ to handle iostreams.  How can this be done?

You don't want to do that.  It would cause C stdio and C++ iostreams
to be buffered separately causing output to be interleaved wrongly.
It would also break the nice stdio/iostreams interface that took so
much work to get right.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: Linux vs. libio

[Ulrich Drepper]

Mark Mitchell <mark@codesourcery.com> writes:

>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.

Wait a minute, why do you think you can change libio?  If something
has to be behave different there must be a different solution found.
You are not with the compiler, don't think you have the right to
change everything.  These things should better be coordinated with me.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Linux vs. libio

[H.J. Lu]

> 
> 
> Folks --
> 
>   Does anyone know how, on Linux, to build the libio (in CVS with GCC)
> in such a way as to be entirely disentangled from the C library?
> (Except in the ways that libio would normally be entangled with the C
> library on any non-glibc system.)
> 
>   In particular, we're working on the new C++ ABI.  That requires some
> changes to libio to deal with different class layouts, etc.  That's
> mostly working, but the stdio functions in glibc have a bad tendency
> to call functions in the modified libio as callbacks, and those
> functions now behave differently than glibc expects.
> 
>   In other words, what I'm looking to do is to have two distinct,
> disentangled, copies of libio: one in libc, to handle stdio, and one
> in libstdc++ to handle iostreams.  How can this be done?
> 

Take a look at how libc5 is handled in libio in gcc. libio in gcc
2.95.2 is quite different from libio in libc5.

-- 
H.J. Lu (hjl@gnu.org)

Linux vs. libio

[Mark Mitchell]

Folks --

  Does anyone know how, on Linux, to build the libio (in CVS with GCC)
in such a way as to be entirely disentangled from the C library?
(Except in the ways that libio would normally be entangled with the C
library on any non-glibc system.)

  In particular, we're working on the new C++ ABI.  That requires some
changes to libio to deal with different class layouts, etc.  That's
mostly working, but the stdio functions in glibc have a bad tendency
to call functions in the modified libio as callbacks, and those
functions now behave differently than glibc expects.

  In other words, what I'm looking to do is to have two distinct,
disentangled, copies of libio: one in libc, to handle stdio, and one
in libstdc++ to handle iostreams.  How can this be done?

  Thanks,

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com