Re: Mixing 32-bit and 64-bit DWARF2/3 sections

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[David Anderson]

James E Wilson <wilson@specifixinc.com> writes:
|SGI Irix6 created the N64 ABI, and they decreed that one must use a 
|(non-standard) 64-bit DWARF format with it.  We are just following the 
|ABI here.  If we change it, then we lose compatibility with Irix6, and 
|the published ABI.

There was no standard 64 bit dwarf format at the time (1992,3,4).
A straight '64 bit address-size and offset' was simple to define
and understand.

|> P.S. It seems that MIPS is the only 64-bit target for which this is a
|>      problem, since all other 64-bit targets (AMD64, UltraSPARC) still
|>      use the 32-bit DWARF format.

|This is because SGI decided that DWARF2 was broken, and tried to fix it 
|by adding 64-bit extensions.  These extensions are only needed if you 
|have more than 4GB of DWARF debug info, and hence is unlikely, but SGI 
|claims that they did hit this limit.  No one else has apparently ever 
|hit the limit.

Allow me to correct that impression.  I don't know that anyone
has exceeded a 4GB dwarf section. Nor did we think DWARF2 was
broken, exactly.  Knowing that the issue of gigantic debug
files was simply never on the table at the DWARF committee
meetings, (there being no need at that point in time), we made
a guess as to what a future DWARFx would have.  Bad guess.
The DWARF committee was either exhausted or dormant (depending
on the date) by this time, so no further changes could
be contemplated there.

This was 1992 or so. 

It was clear from past experience was that *someone would*
exceed 32 bits of offset.  And when it happened, the
restriction was going to seem arbitrary and would be painful to
correct.

Every limit in mdebug had been hit by that time, and being
unable to correct those problems was...painful.  We'd only felt
able to correct an mdebug limit once (back around 1989) and
that binary incompatibility was --- painful to deal with.  (In
that case MIPS [as a separate company] broke the ABI to fix an
mdebug limit and we went along as that was the only real
choice.)

|Unfortunately, the way that SGI extended DWARF2 caused problems, as it 
|resulted in ambiguous DWARF info, so the DWARF3 committee defined a 
|different way that is self descriptive.  But since this happened after 
|Irix6 had already been released, and SGI could not break backwards 
|compatibility with Irix6, N64 still uses the non-standard 64-bit DWARF 
|format that SGI invented.  And gcc uses this non-standard 64-bit DWARF 
|format for all mips64 targets, for consistency, and to conform with the ABI.

At the time there was only one 64bit MIPS, and no 64bit offset DWARF.
[And no other 64-bit ELF users? (I vaguely recall.).  No 
other DWARF2 users at that time.]
We at SGI felt obliged to do something to avoid the limits in 32bit
offset DWARF2.  Our choice did not really meet the needs and desires of
today's community thus DWARF3 has a different approach to 64bit offsets.

Daniel Berlin <dberlin@dberlin.org> writes:
[...]
|In short, there is no limitation in dwarf3 that SGI is solving.
|It's just a matter of them not solving it in a backwards compatible way, 
|and everyone else paying the consequences :)

Well, we associate dwarf-offset with elf-offset.
So 64-bit-elf-offset means 64-bit-dwarf offset.
It seemed perfectly compatible then.
Our solution was for a limitation in DWARF2, and only DWARF2 existed
at the time.

Jim Wilson again:
>I don't think we should be worried about use of 32-bit DWARF2 debug
>info.  We are unlikely to hit the limits anytime soon.  And if we do hit
>the limits, then we will likely have other more serious problems.

The limits will be hit, likely before long.
The problems will be seen only in a tiny subset of applications.
Since 13TB ram-memory machines exist today, a 5GB .debug_info
does not seem that large. Admittedly such a size is likely avoidable
by using shared-libraries.

On IRIX the only appropriate 64-bit DWARF2 is
the IRIX definition.    Beyond that I'm not going...

David Anderson
[PS: Opinions my own, not those of SGI. History is as I recall what
happened at the time.]

Mixing 32-bit and 64-bit DWARF2/3 sections

[Mark Kettenis]

Currently GDB barfs when you use it on an executable that contains
compilation units that mix 32-bit and 64-bit DWARF sections.  Looking
at the (latest?) DWARF3 draft it doesn't seem to be unreasonable that
it does so:

  "The 32-bit and 64-bit DWARF format conventions must not be
   intermixed within a single compilation unit."

The way things are structured now, it is easy to end up with a
configuration where this happens since some DWARF sections are
generated by GCC and others by gas, i.e. if GCC is configured to
generate 32-bit DWARF, and gas will generate 64-bit dwarf.  This is
currently the case for the (not yet contributed) GCC configuration
that's currently used by OpenBSD/mips64.  Now this can be easily fixed
by changing either the GCC config or gas, but the DWARF3 draft also
says:

  "It is expected that DWARF producing compilers will not use the
   64-bit format by default. In most cases, the division of even very
   large applications into a number of executable and shared objects
   will suffice to assure that the DWARF sections within each
   individual linked object are less than 4 GBytes in size. However,
   for those cases where needed, the 64-format allows the unusual case
   to be handled as well. Even in this case, it is expected that only
   application supplied objects will need be compiled using the 64-bit
   format; separate 32-bit format versions of system supplied shared
   executable libraries can still be used."

This argues that what's currently done in gas and GCC for most 64-bit
MIPS targets is wrong, since these use the 64-bit DWARF format
unconditionally for the n64 ABI.

Fixing this would be a tricky though, since doing it requires some
coordination between GCC and binutils.  We could add a switch to gas
to tell it to generate 64-bit DWARF format.  But even then GCC will
probably need to support old gas versions that generate 64-bit format
by default, adding more config goo to GCC.  So I think the existing
MIPS targets should probably keep doing what they do now.

However, I could make OpenBSD/mips64 do the right thing, making both
GCC and gas generate 32-bit DWARF by default.  What do the MIPS people
think about that?

Cheers,

Mark

P.S. It seems that MIPS is the only 64-bit target for which this is a
     problem, since all other 64-bit targets (AMD64, UltraSPARC) still
     use the 32-bit DWARF format.

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Daniel Jacobowitz]

On Sun, Nov 07, 2004 at 03:28:07PM +0100, Mark Kettenis wrote:
> Currently GDB barfs when you use it on an executable that contains
> compilation units that mix 32-bit and 64-bit DWARF sections.  Looking
> at the (latest?) DWARF3 draft it doesn't seem to be unreasonable that
> it does so:
> 
>   "The 32-bit and 64-bit DWARF format conventions must not be
>    intermixed within a single compilation unit."

I'd read that as saying that GDB should support what you describe. 
This isn't mixing them within a single compilation unit.  What falls
down?

-- 
Daniel Jacobowitz

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Mark Kettenis]

   Date: Sun, 7 Nov 2004 10:31:44 -0500
   From: Daniel Jacobowitz <drow@false.org>

   On Sun, Nov 07, 2004 at 03:28:07PM +0100, Mark Kettenis wrote:
   > Currently GDB barfs when you use it on an executable that contains
   > compilation units that mix 32-bit and 64-bit DWARF sections.  Looking
   > at the (latest?) DWARF3 draft it doesn't seem to be unreasonable that
   > it does so:
   > 
   >   "The 32-bit and 64-bit DWARF format conventions must not be
   >    intermixed within a single compilation unit."

   I'd read that as saying that GDB should support what you describe. 
   This isn't mixing them within a single compilation unit.  What falls
   down?

No I'm really talking about mixing 32-bit and 64-bit DWARF format
within the same compilation unit.  In the case of OpenBSD/mips64 GCC
generates a 32-bit .debug_info section and .file directives that
contain the line info.  These .file directives are then converted by
gas into 64-bit .debug_line sections.

When GDB reads these it dumps core.  I'm working on patches to fix
this, but I'm not yet completely satisfied with them.

Mark

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Daniel Jacobowitz]

On Sun, Nov 07, 2004 at 05:28:41PM +0100, Mark Kettenis wrote:
>    Date: Sun, 7 Nov 2004 10:31:44 -0500
>    From: Daniel Jacobowitz <drow@false.org>
> 
>    On Sun, Nov 07, 2004 at 03:28:07PM +0100, Mark Kettenis wrote:
>    > Currently GDB barfs when you use it on an executable that contains
>    > compilation units that mix 32-bit and 64-bit DWARF sections.  Looking
>    > at the (latest?) DWARF3 draft it doesn't seem to be unreasonable that
>    > it does so:
>    > 
>    >   "The 32-bit and 64-bit DWARF format conventions must not be
>    >    intermixed within a single compilation unit."
> 
>    I'd read that as saying that GDB should support what you describe. 
>    This isn't mixing them within a single compilation unit.  What falls
>    down?
> 
> No I'm really talking about mixing 32-bit and 64-bit DWARF format
> within the same compilation unit.  In the case of OpenBSD/mips64 GCC
> generates a 32-bit .debug_info section and .file directives that
> contain the line info.  These .file directives are then converted by
> gas into 64-bit .debug_line sections.
> 
> When GDB reads these it dumps core.  I'm working on patches to fix
> this, but I'm not yet completely satisfied with them.

Ah!  I understand now, thanks.

I think GDB should complain noisily about this.  One nice fix to this
in gas would be for it to detect the format of .debug_info and match it
when generating .debug_line.  Another would be to add command line
options for 32-bit and 64-bit .debug_line and make GCC instruct it on
which to use - that's a bit more complicated to implement.

-- 
Daniel Jacobowitz

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[James E Wilson]

Mark Kettenis wrote:
> This argues that what's currently done in gas and GCC for most 64-bit
> MIPS targets is wrong, since these use the 64-bit DWARF format
> unconditionally for the n64 ABI.

SGI Irix6 created the N64 ABI, and they decreed that one must use a 
(non-standard) 64-bit DWARF format with it.  We are just following the 
ABI here.  If we change it, then we lose compatibility with Irix6, and 
the published ABI.

> P.S. It seems that MIPS is the only 64-bit target for which this is a
>      problem, since all other 64-bit targets (AMD64, UltraSPARC) still
>      use the 32-bit DWARF format.

This is because SGI decided that DWARF2 was broken, and tried to fix it 
by adding 64-bit extensions.  These extensions are only needed if you 
have more than 4GB of DWARF debug info, and hence is unlikely, but SGI 
claims that they did hit this limit.  No one else has apparently ever 
hit the limit.

Unfortunately, the way that SGI extended DWARF2 caused problems, as it 
resulted in ambiguous DWARF info, so the DWARF3 committee defined a 
different way that is self descriptive.  But since this happened after 
Irix6 had already been released, and SGI could not break backwards 
compatibility with Irix6, N64 still uses the non-standard 64-bit DWARF 
format that SGI invented.  And gcc uses this non-standard 64-bit DWARF 
format for all mips64 targets, for consistency, and to conform with the ABI.
-- 
Jim Wilson, GNU Tools Support, http://www.SpecifixInc.com

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Mark Kettenis]

   Date: Mon, 08 Nov 2004 12:54:30 -0800
   From: James E Wilson <wilson@specifixinc.com>

   Mark Kettenis wrote:
   > This argues that what's currently done in gas and GCC for most 64-bit
   > MIPS targets is wrong, since these use the 64-bit DWARF format
   > unconditionally for the n64 ABI.

   SGI Irix6 created the N64 ABI, and they decreed that one must use a 
   (non-standard) 64-bit DWARF format with it.  We are just following the 
   ABI here.  If we change it, then we lose compatibility with Irix6, and 
   the published ABI.

Yes I left out the non-standard SGI 64-bit DWARF format.  Indeed it
makes sense to maintain compatibility with the vendor tools if
possible.  So for the IRIX n64 ABI the GNU toolchain should generate
the non-standard 64-bit DWARF.

The situation is different though for other 64-bit MIPS systems.  The
n64 ABI for Linux and OpenBSD is already incompatible with IRIX
(bfd_elf64_bigmips_vec vs. bfd_elf64_tradbigmips_vec).  On these
systems, gas currently generates the new DWARF3 64-bit format.  I
merely want to argue that it probably would have been better if the
default had been 32-bit for Linux.

   > P.S. It seems that MIPS is the only 64-bit target for which this is a
   >      problem, since all other 64-bit targets (AMD64, UltraSPARC) still
   >      use the 32-bit DWARF format.

   This is because SGI decided that DWARF2 was broken, and tried to fix it 
   by adding 64-bit extensions.  These extensions are only needed if you 
   have more than 4GB of DWARF debug info, and hence is unlikely, but SGI 
   claims that they did hit this limit.  No one else has apparently ever 
   hit the limit.

   Unfortunately, the way that SGI extended DWARF2 caused problems, as it 
   resulted in ambiguous DWARF info, so the DWARF3 committee defined a 
   different way that is self descriptive.  But since this happened after 
   Irix6 had already been released, and SGI could not break backwards 
   compatibility with Irix6, N64 still uses the non-standard 64-bit DWARF 
   format that SGI invented.  And gcc uses this non-standard 64-bit DWARF 
   format for all mips64 targets, for consistency, and to conform with the ABI.

If this is true, there's a real problem in 64-bit MIPS Linux now,
since gas clearly uses the non-standard 64-bit format only on IRIX
according to this fragment from gas/config/tc-mips.c:

enum dwarf2_format
mips_dwarf2_format (void)
{
  if (mips_abi == N64_ABI)
    {
#ifdef TE_IRIX
      return dwarf2_format_64bit_irix;
#else
      return dwarf2_format_64bit;
#endif
    }
  else
    return dwarf2_format_32bit;
}

On the bright side, if all non-IRIX 64-bit MIPS targets are currently
broken, this gives us the opportunity to fix things the "right" way by
using standard 32-bit DWARF2 by default.

Mark

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[James E Wilson]

On Mon, 2004-11-08 at 14:04, Mark Kettenis wrote:
> systems, gas currently generates the new DWARF3 64-bit format.  I
> merely want to argue that it probably would have been better if the
> default had been 32-bit for Linux.

It would be reasonable to try to use standard DWARF for non-Irix
systems, but the ambiguous SGI DWARF format complicates this.

With the SGI DWARF format, you can't tell whether you have 32-bit or
64-bit debug info by looking at the debug info alone.  You need external
info.  Currently, this is based on the ABI.  If we have N32 code, then
we have 32-bit DWARF always.  If we have N64 code, then we have
SGI-64-bit DWARF always.  (I see that neither bfd nor gdb actually do it
this way, but they are supposed to.)

In order to change this, we will need some other header flag to indicate
whether we are following the SGI scheme or the GNU scheme.  If we have
an OS field that is set consistently, then we could use that, but I as
skeptical whether this is safe in all cases.  Not all operating systems
have their own OS flags defined, especially for embedded systems.  There
is also a backwards compatibility problem.  That means we need a new
flag.  We need assembler support to set it, and an assembler directive
or option so that gcc can tell the assembler whether or not to set it. 
So we need an ELF header change, along with gcc, binutils, and gdb
patches to deal with it.  It isn't clear to me whether the benefits of
this change are worth the amount of effort needed to implement it.

Anyways, I think it is a reasonable goal, just not sure if it is worth
the trouble it will cause.

Besides the technical details, you will also need to get political buy
in from affected parties for an ABI change, which means mainly the
GNU/Linux mips64 folks.

I'd suggest that the easiest solution here is to fix gas.  Gas should
always use dwarf2_format_64bit_irix with the N64 ABI, regardless of
whether the target is Irix.  This will fix your debugging problems,
though it won't give the debug info format you want.

Checking, I see that the binutils patch came from here:
	http://sources.redhat.com/ml/binutils/2003-01/msg00442.html
This is a thread with Alexandre Oliva trying to fix some problems with
the SGI-64-bit DWARF2 format, and Daniel Jacobowitz questioning why we
are using a non-standard format for non-Irix targets, and Alex made this
change to try to make Daniel happy.  Unfortunately, this causes your
current problem, as it makes gas incompatible with gcc.  I'd suggest
reverting this gas patch to solve your problem as a short term fix. 
Incidentally, this thread contains some info explaining why Alex thought
it was a good idea to be compatible with Irix here, as it made some
things easier.  The long term fix is a new elf header flag, and gcc,
binutils, and gdb changes to set and use it.  Reverting the gas patch is
far simpler.
-- 
Jim Wilson, GNU Tools Support, http://www.SpecifixInc.com

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Thiemo Seufer]

James E Wilson wrote:
[snip]
> Besides the technical details, you will also need to get political buy
> in from affected parties for an ABI change, which means mainly the
> GNU/Linux mips64 folks.

FWIW, I doubt going back unconditionally to a 32bit format is the right
thing to do for a fully 64bit ABI. Applications tend to grow further,
as does the accompanying debug info, so the 32bit limit comes closer.


Thiemo

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Daniel Berlin]

On Tue, 9 Nov 2004, Thiemo Seufer wrote:

> James E Wilson wrote:
> [snip]
>> Besides the technical details, you will also need to get political buy
>> in from affected parties for an ABI change, which means mainly the
>> GNU/Linux mips64 folks.
>
> FWIW, I doubt going back unconditionally to a 32bit format is the right
> thing to do for a fully 64bit ABI. Applications tend to grow further,
> as does the accompanying debug info, so the 32bit limit comes closer.

DWARF3, which is what gcc emits, is 32 bit by "default", but if it 
became necessary, it (DWARF3) supports 64 bit sizes/offsets/etc and can 
output 
them.

The main difference between SGI's 64 bit-only dwarf2, and DWARF3's 
64bit dwarf2 is that DWARF3 allows you to mix 32 bit and 64 bit 
compilation units ( you just can't mix 32 bit and 64 bit *in* a single 
compilation unit).
SGI doesn't allow you to do this.

DWARF3 accomplishes this by using a 32 bit 0xffffffff as the 
initial length in the compilation unit to signal that this is really 64 
bit, which is followed by a 64 bit real length.

SGI does no such thing, they simply use a 64 bit initial length, which 
means you don't know whether you have 32 bit or 64 bit unless the ABI 
tells you it must be one or the other.

In short, there is no limitation in dwarf3 that SGI is solving.
It's just a matter of them not solving it in a backwards compatible way, 
and everyone else paying the consequences :)

--Dan

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[James E Wilson]

On Mon, 2004-11-08 at 15:37, Thiemo Seufer wrote:
> FWIW, I doubt going back unconditionally to a 32bit format is the right
> thing to do for a fully 64bit ABI. Applications tend to grow further,
> as does the accompanying debug info, so the 32bit limit comes closer.

DWARF2 debug info has a field to indicate pointer size, and uses this
for addresses.  The only limit to DWARF2 debug info is that cross
references within the DWARF2 debug info are specified by 32-bit fields. 
Thus, in order to exceed the 32-bit limit, one of the DWARF2 sections
must be larger than 4GB.  Since the DWARF2 info is spread across
multiple sections, it would be possible to have 36GB of DWARF2 debug
info, and still not exceed the 32-bit limit, if none of the individual
sections are larger than 4GB.  You can have a 1TB application, and still
not exceed the 32-bit limit, if none of the individual DWARF2 debug
section are larger than 4GB.

I don't think we should be worried about use of 32-bit DWARF2 debug
info.  We are unlikely to hit the limits anytime soon.  And if we do hit
the limits, then we will likely have other more serious problems.
-- 
Jim Wilson, GNU Tools Support, http://www.SpecifixInc.com

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Daniel Jacobowitz]

On Mon, Nov 08, 2004 at 12:54:30PM -0800, James E Wilson wrote:
> Unfortunately, the way that SGI extended DWARF2 caused problems, as it 
> resulted in ambiguous DWARF info, so the DWARF3 committee defined a 
> different way that is self descriptive.  But since this happened after 
> Irix6 had already been released, and SGI could not break backwards 
> compatibility with Irix6, N64 still uses the non-standard 64-bit DWARF 
> format that SGI invented.  And gcc uses this non-standard 64-bit DWARF 
> format for all mips64 targets, for consistency, and to conform with the ABI.

This is not correct.

% grep DWARF_INITIAL_LENGTH_SIZE *
iris.h:#define DWARF_INITIAL_LENGTH_SIZE DWARF_OFFSET_SIZE
%

Only Irix 5 and Irix 6 use the SGI DWARF format.  mips64-linux has been
using the standard DWARF3 format for a long while.

-- 
Daniel Jacobowitz

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[James E Wilson]

On Mon, 2004-11-08 at 15:45, Daniel Jacobowitz wrote:
> Only Irix 5 and Irix 6 use the SGI DWARF format.  mips64-linux has been
> using the standard DWARF3 format for a long while.

Ah, you are right.  I was misreading the gcc sources I was looking at.

So the mips64-linux port defaults to the standard DWARF3 64-bit dwarf
debug info format, even though there seems to be no reason to use the
64-bit DWARF3 debug format here.  If we aren't going to be compatible
with SGI, then we may as well emit 32-bit DWARF3, and only emit 64-bit
DWARF3 if we need it, which will be very unlikely.  This is what all
other targets do.  The unnecessary use of the 64-bit debug info here is
making debug info, and object files, larger than necessary.  Since you
are already using standard DWARF3 here, changing can perhaps be done
without creating a visible ABI change.

It is simpler to leave things the way they already are though, and ask
mips64-openbsd to conform to the mips64-linux choices.
-- 
Jim Wilson, GNU Tools Support, http://www.SpecifixInc.com

Re: Mixing 32-bit and 64-bit DWARF2/3 sections

[Dean Luick]

On Sun, Nov 07, 2004 at 03:28:07PM +0100, Mark Kettenis wrote:
> P.S. It seems that MIPS is the only 64-bit target for which this is a
>      problem, since all other 64-bit targets (AMD64, UltraSPARC) still
>      use the 32-bit DWARF format.

Not quite.  All versions of the Cray X1 compiler will produce 64 bit
DWARF unconditionally.  Versions 5.2 and earlier of the Cray X1 compiler
produce the SGI "special" version of 64-bit dwarf.  Versions 5.3 and later
follow the DWARF 3 method of differentiating 32- and 64-bit.

Reason for Cray being the same as SGI:  Cray was a part of SGI until it was
sold to Terra in 2000.

However, since there currently is no support in binutils/GDB for the Cray X1,
the point is rather moot.  :-)


Dean