64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

[Jan Kratochvil]

Hello,

original problem - GCC produces working >4GB data types code while GDB cannot
handle them:
	[PATCH] Expand bitpos to LONGEST to allow access to large offsets within a struct
	http://sourceware.org/ml/gdb-patches/2012-02/msg00403.html

There should be checked in soon thanks to Siddhesh Poyarekar an extension of
TYPE_LENGTH from 'unsigned int' to ULONGEST:
	[PATCH 0/4] bitpos expansion summary reloaded
	http://sourceware.org/ml/gdb-patches/2012-09/msg00628.html

Anything checked-in since the following commit would be great to no longer
break this goal to handle it correctly:
	http://sourceware.org/ml/gdb-cvs/2012-09/msg00161.html
	commit cec90d9d386f57f116e114c50e4287281420f531
	Author: siddhesh <siddhesh>
	Date:   Thu Sep 27 10:39:58 2012 +0000
		* amd64-tdep.c (amd64_return_value): Revert previous change
		that used TYPE_LENGTH directly.
	[...]
		* vax-tdep.c (vax_return_value): Likewise.

So far GDB was freely using 'int' for any inferior sizes.  This is no longer
compatible, use always wide enough types:

(1) signed vs. unsigned types do not matter for inferior data types sizes.
    Such as LONGEST vs. ULONGEST or ssize_t vs. size_t.

    GDB will never successfully allocate more than 2GB on 32-bit host (size_t
    max larger than ssize_t).  And it does not make sense to consider anyhow
    sizes above 9 quintillion (ULONGEST max larger than LONGEST on any host or
    size_t larger than ssize_t on 64-bit hosts).

(2) Use the smallest valid type, therefore use size_t/ssize_t for objects
    present in host GDB memory - not LONGEST/ULONGEST.  On 32-bit hosts it
    cheaper and also IMO the code is more readable with appropriate types.
    BTW for size_t/ssize_t format string is "%z".

    Also use CORE_ADDR where appropriate, GDB was also using sometimes
    incorrectly LONGEST/ULONGEST instead.
    BTW for CORE_ADDR format string is "%s", paddress (gdbarch, X).

    BTW for LONGEST/ULONGEST format string
    is "%s", plongest (X) or "%s", pulongest (X) accordingly.

    Never use long / unsigned long (it is host platform dependent).

(3) For allocating inferior data in host GDB memory, therefore assigning
    LONGEST/ULONGEST (type of TYPE_LENGTH) to size_t/ssize_t variable:
    One needs to manually check the value fits in.
    Call ulongest_fits_host_or_error for that.

    It will ensure the size <= SIZE_MAX / 8 so even small additions after the
    check will not overflow the type before it gets passed to xmalloc.

(4) Copying >2GB inferior data to host memory is probably unreal.  In reality
    GDB should copy the data on-demand.  But that is a future patch outside of
    the scope of this patchset.
    
    IIUC such patch will be at least easier to write when the code already
    knows the valid 64-bit size of the inferior object.  Still it is currently
    more correct to "lock up" GDB (interruptible by CTRL-C) than to respond to
    user with invalid/shortened data without printing any error.

(5) Patched GDB expects only these inferior types may have >2GB size:
    	TYPE_CODE_ARRAY, TYPE_CODE_STRUCT, TYPE_CODE_UNION
    And of course where applicable:
    	TYPE_CODE_TYPEDEF
    While a compiler may create
    	 <1><57>: Abbrev Number: 3 (DW_TAG_base_type)
    	    <58>   DW_AT_byte_size   : 0x123456789
    	    <59>   DW_AT_encoding    : 5        (signed)
    	    <5a>   DW_AT_name        : veryhugeint
    it is OK GDB will not internally handle it correctly, not giving any error.
    
    There could be a new patch to reject such types in dwarf2read.c on read-in.

(6) If a function is called only for small inferior types (scalars) then it is
    OK to have there / keep there 'int' type for the inferior type size.
    Callers have to ensure array/struct/union type cannot be passed there.

    If a function does
    	gdb_assert (length < 16);
    then sure it should not happen - it is an assert.  But still one needs to
    keep the 'length' parameter as LONGEST/ULONGEST so that the assert makes
    sense.


Thanks,
Jan

Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

[Sergio Durigan Junior]

On Friday, September 28 2012, Jan Kratochvil wrote:

> Hello,

Thanks for the message and explanations.  The way I see it, they are
instructions on how a GDB developer should proceed in certain
situations.  May I suggest that a wiki page be created to contain those
instructions?  Otherwise, I am afraid this e-mail will get lost in the
archives and nobody will notice it in a few weeks.

I can create the page if you agree with it.

Thanks,

-- 
Sergio

Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

[Jan Kratochvil]

On Fri, 28 Sep 2012 22:23:10 +0200, Sergio Durigan Junior wrote:
> May I suggest that a wiki page be created to contain those instructions?

There are now both
	http://sourceware.org/gdb/wiki/JoelsCodingStyleCheatSheet
and
	http://sourceware.org/gdb/current/onlinedocs/gdbint/Coding-Standards.html
(the former explains it)

I believe these notes belong more to gdbint.texinfo.


> Otherwise, I am afraid this e-mail will get lost in the
> archives and nobody will notice it in a few weeks.

Its purpose was to act first as a discussion subject before some final form
which I did not think much about so far.  While it may be clear I have found
there are some disagreements so possible changes to the rules could have also
effect on the final / largest patchset check-in.


> I can create the page if you agree with it.

But you are right it would be best, after it gets approved (at least by
a timeout) it would be great to move it to gdbint.texinfo, or I could.


Thanks,
Jan

Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

[Jan Kratochvil]

On Fri, 28 Sep 2012 19:32:29 +0200, Jan Kratochvil wrote:
> (5) Patched GDB expects only these inferior types may have >2GB size:
>     	TYPE_CODE_ARRAY, TYPE_CODE_STRUCT, TYPE_CODE_UNION
>     And of course where applicable:
>     	TYPE_CODE_TYPEDEF

TYPE_CODE_STRING technically can be >2GB but I do not think it makes much
sense in practice, also current GDB cannot handle it well anyway, I was not
strictly ensuring its >2GB handling is correct.


Jan

Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST

[Joel Brobecker]

>     BTW for size_t/ssize_t format string is "%z".

Unfortunately, that's C99... Not sure what the best approach would
be - I think people typically cast to long or unsigned long and
then use the us %l or %ul.

-- 
Joel

Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Jan Kratochvil]

On Sun, 30 Sep 2012 18:56:30 +0200, Joel Brobecker wrote:
> >     BTW for size_t/ssize_t format string is "%z".
> 
> Unfortunately, that's C99... Not sure what the best approach would
> be - I think people typically cast to long or unsigned long and
> then use the us %l or %ul.

this is repeating issue, GDB is not C90 compatible despite being claimed so:
	Re: [no-commit-intention] Naive unnamed fields for main_type [...]
	http://sourceware.org/ml/gdb-patches/2012-02/msg00146.html

GDB has not been kept compatible even with Solaris which is a very live OS:
	Various build-related fixes for solaris
	http://sourceware.org/ml/gdb-patches/2012-09/msg00404.html

Summary of the first thread (authorship of the statements is in mails above):

Even GCC does not fully support C99 ( http://gcc.gnu.org/c99status.html )
but all compilers in use support the important parts of C99, incl. gcc-2.9x.
As an example anonymous unions are supported by MS Visual Studio 6.0 (1998),
  http://msdn.microsoft.com/en-us/library/aa270923%28v=vs.60%29.aspx
'long long' is supported at least by MS Visual Studio 2005:
  http://msdn.microsoft.com/en-us/library/s3f49ktz%28v=vs.80%29.aspx
I have no idea about HP cc etc. but I believe those in use also support it.
Moreover GNU policy is OK with even C99 (probably its GCC subset).

And issues like insufficient system vasprintf can be easily fixed by adding
vasprintf from gnulib.

If there pops up some incompatibility it can be easily fixed / reverted.
Keeping compatibility with ghost systems nobody knows whether they still exist
I do not find useful.  Various fixed incl. 'Various build-related fixes for
solaris' prove the compatibility with OSes not in use is not out of the box.
Just there should be kept some viable way how to fix it up when such
compatibility issue pops up.


Therefore I believe "%z" is OK, it would be nice to check it with several
major non-GNU systems whether gnulib vasprintf should be already included.


Thanks,
Jan

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Pedro Alves]

On 10/01/2012 05:48 PM, Jan Kratochvil wrote:

> Therefore I believe "%z" is OK, it would be nice to check it with several
> major non-GNU systems whether gnulib vasprintf should be already included.

Older mingw versions would be a host I recall whose libc didn't
use to support %z until a couple years ago.  I believe Eli (like many, due to
some technical limitations of gcc 4.x) for example stills uses a mingw with a
3.x gcc.  Not sure whether people are combining newer mingw runtime releases with
the 3.x based compilers.

-- 
Pedro Alves

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Eli Zaretskii]

> Date: Mon, 1 Oct 2012 18:48:33 +0200
> From: Jan Kratochvil <jan.kratochvil@redhat.com>
> Cc: gdb@sourceware.org, Siddhesh Poyarekar <siddhesh@redhat.com>
> 
> On Sun, 30 Sep 2012 18:56:30 +0200, Joel Brobecker wrote:
> > >     BTW for size_t/ssize_t format string is "%z".
> > 
> > Unfortunately, that's C99... Not sure what the best approach would
> > be - I think people typically cast to long or unsigned long and
> > then use the us %l or %ul.
> 
> this is repeating issue, GDB is not C90 compatible despite being claimed so:
> 	Re: [no-commit-intention] Naive unnamed fields for main_type [...]
> 	http://sourceware.org/ml/gdb-patches/2012-02/msg00146.html
> 
> GDB has not been kept compatible even with Solaris which is a very live OS:
> 	Various build-related fixes for solaris
> 	http://sourceware.org/ml/gdb-patches/2012-09/msg00404.html
> 
> Summary of the first thread (authorship of the statements is in mails above):
> 
> Even GCC does not fully support C99 ( http://gcc.gnu.org/c99status.html )
> but all compilers in use support the important parts of C99, incl. gcc-2.9x.
> As an example anonymous unions are supported by MS Visual Studio 6.0 (1998),
>   http://msdn.microsoft.com/en-us/library/aa270923%28v=vs.60%29.aspx
> 'long long' is supported at least by MS Visual Studio 2005:
>   http://msdn.microsoft.com/en-us/library/s3f49ktz%28v=vs.80%29.aspx

It is not a good idea to judge about overall C99 acceptance by looking
at its select features.  Some features are very widespread, others
much less.

Therefore, the fact that MSVC supports anonymous unions says nothing
about its support of %z.

Moreover, at least regarding the MS platforms and MinGW (not MSVC),
there are 2 separate components involved.  One is GCC, which could be
very modern and know about C99 and %z, the other is the runtime
library, which is not redistributable, and therefore one uses whatever
came with the OS out of the box.  For example, a Windows XP system
will generally have runtime that goes back to Studio 2003 (VC version
7.1), and Windows 7 will use VC v9.0, I think, from Studio 2008.
These are quite old.  And support for %z comes from the library, of
course.

> Therefore I believe "%z" is OK, it would be nice to check it with several
> major non-GNU systems whether gnulib vasprintf should be already included.

FWIW, even the latest Studio 2012 doesn't yet support %z, according to
this:

  http://msdn.microsoft.com/en-us/library/hf4y5e3w%28v=vs.110%29.aspx

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Eli Zaretskii]

> Date: Mon, 01 Oct 2012 18:18:37 +0100
> From: Pedro Alves <palves@redhat.com>
> CC: Joel Brobecker <brobecker@adacore.com>, gdb@sourceware.org,
>         Siddhesh Poyarekar <siddhesh@redhat.com>, Eli Zaretskii <eliz@gnu.org>
> 
> On 10/01/2012 05:48 PM, Jan Kratochvil wrote:
> 
> > Therefore I believe "%z" is OK, it would be nice to check it with several
> > major non-GNU systems whether gnulib vasprintf should be already included.
> 
> Older mingw versions would be a host I recall whose libc didn't
> use to support %z until a couple years ago.

I don't think MinGW can support %z even today, because the CRT DLL
doesn't.  See my other message.

> I believe Eli (like many, due to some technical limitations of gcc
> 4.x) for example stills uses a mingw with a 3.x gcc.

That's true, but the main issue here is the library, not the compiler.

> Not sure whether people are combining newer mingw runtime releases
> with the 3.x based compilers.

MinGW runtime does not replace the format conversion engine, it uses
the MS provided one, AFAIK.  It does augment the MS runtime with
several functions of the printf family, but AFAIK they do not include
replacement of the format-conversion code.  If you know otherwise,
please tell which MinGW source file includes this replacement.

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Jan Kratochvil]

On Mon, 01 Oct 2012 19:32:01 +0200, Eli Zaretskii wrote:
> FWIW, even the latest Studio 2012 doesn't yet support %z, according to
> this:
> 
>   http://msdn.microsoft.com/en-us/library/hf4y5e3w%28v=vs.110%29.aspx

I would prefer a runtime test as not everything may be documented but OK,
I see gnulib vasprintf check-in should go along the future %z check-in.



Thanks,
Jan

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Joel Brobecker]

> I see gnulib vasprintf check-in should go along the future %z check-in.

I am not opposed to this, but we should make sure that gnulib's check
for vasprintf does verify availability of %z. I'm not sure that it is
the case today.

-- 
Joel

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Pedro Alves]

On 10/01/2012 06:38 PM, Eli Zaretskii wrote:
>> From: Pedro Alves <palves@redhat.com>

>> Older mingw versions would be a host I recall whose libc didn't
>> use to support %z until a couple years ago.

It used to last I looked, but not by default.   See below.

> 
> I don't think MinGW can support %z even today, because the CRT DLL
> doesn't.  See my other message.
> 
>> I believe Eli (like many, due to some technical limitations of gcc
>> 4.x) for example stills uses a mingw with a 3.x gcc.
> 
> That's true, but the main issue here is the library, not the compiler.
> 
>> Not sure whether people are combining newer mingw runtime releases
>> with the 3.x based compilers.
> 
> MinGW runtime does not replace the format conversion engine, it uses
> the MS provided one, AFAIK.  It does augment the MS runtime with
> several functions of the printf family, but AFAIK they do not include
> replacement of the format-conversion code.  If you know otherwise,
> please tell which MinGW source file includes this replacement.

In the old mingw sources copy I have here (2009), I see under
mingwex/stdio/ a printf.c file, which provides the __mingw_printf replacement.
The format conversion engine is in pformat.c, and it does implement '%z'
(look for "case 'z':").

See <http://sourceforge.net/project/shownotes.php?release_id=24832>
for how to enable the replacements.  E.g., #define _GNU_SOURCE
or __USE_MINGW_ANSI_STDIO to 1 should be enough (it defines printf to
__mingw_printf).

I tried looking for the current sources, but the mingw.org frontpage has a news
item mentioning that the sources have moved to git, without giving a pointer to
where the repository is...  Then, after wasting some minutes trying to find
a pointer (and failing; in the end google found it)
I found that http://sourceforge.net/projects/mingw/develop points
at "git clone git://mingw.git.sourceforge.net/gitroot/mingw/mingw",
but cloning that yields an empty repo.  WTH?  Is the project actively
trying to hide itself in some dark internet corner?

-- 
Pedro Alves

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Jan Kratochvil]

On Mon, 01 Oct 2012 19:57:36 +0200, Joel Brobecker wrote:
> > I see gnulib vasprintf check-in should go along the future %z check-in.
> 
> I am not opposed to this, but we should make sure that gnulib's check
> for vasprintf does verify availability of %z. I'm not sure that it is
> the case today.

That's true gnulib does not check for %z.  But whole .microsoft.com does not
mention vasprintf so I believe it would get substituted by gnulib there.

Currently it probably gets substituted by libiberty/_doprnt.c which does not
support %z so the vasprintf inclusion by gnulib is still needed.


Thanks,
Jan

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Joel Brobecker]

> That's true gnulib does not check for %z.  But whole .microsoft.com does not
> mention vasprintf so I believe it would get substituted by gnulib there.

Microsoft is only one example of such systems, however. We cannot use
this example to say that the %z check is unnecessary for all systems.

-- 
Joel

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Jan Kratochvil]

On Mon, 01 Oct 2012 20:13:38 +0200, Joel Brobecker wrote:
> > That's true gnulib does not check for %z.  But whole .microsoft.com does not
> > mention vasprintf so I believe it would get substituted by gnulib there.
> 
> Microsoft is only one example of such systems, however. We cannot use
> this example to say that the %z check is unnecessary for all systems.

vasprintf is a GNU extension.  glibc implemented vasprintf approx. on:
	96aa2d94a2355cdc55c96e808d14a0e7f2ebe77d Date: Mon Nov 20 03:48:11 1995 +0000
while %z glibc implemented approx. on:
	e852e889444a8bf27f3e5075d064e9922b38e7e2 Date: Tue Jul 28 16:26:04 1998 +0000

I do not say there cannot exist an OS with vasprintf not supporting %z but it
seems improbable enough to me to handle it specially before such OS is found.

As mentioned in the original mail it can be still easily fixed up after
a first user of such OS bugreports it.


Thanks,
Jan

Re: Old OSes compatibility [Re: 64-bit (>4GB) inferior data types rules; TYPE_LENGTH: unsigned -> ULONGEST]

[Eli Zaretskii]

> Date: Mon, 01 Oct 2012 19:01:19 +0100
> From: Pedro Alves <palves@redhat.com>
> CC: jan.kratochvil@redhat.com, brobecker@adacore.com, gdb@sourceware.org,
>         siddhesh@redhat.com
> 
> In the old mingw sources copy I have here (2009), I see under
> mingwex/stdio/ a printf.c file, which provides the __mingw_printf replacement.
> The format conversion engine is in pformat.c, and it does implement '%z'
> (look for "case 'z':").
> 
> See <http://sourceforge.net/project/shownotes.php?release_id=24832>
> for how to enable the replacements.  E.g., #define _GNU_SOURCE
> or __USE_MINGW_ANSI_STDIO to 1 should be enough (it defines printf to
> __mingw_printf).

In that case, perhaps GDB should use __USE_MINGW_ANSI_STDIO by default
when building for MinGW.

> I tried looking for the current sources, but the mingw.org frontpage has a news
> item mentioning that the sources have moved to git, without giving a pointer to
> where the repository is...  Then, after wasting some minutes trying to find
> a pointer (and failing; in the end google found it)
> I found that http://sourceforge.net/projects/mingw/develop points
> at "git clone git://mingw.git.sourceforge.net/gitroot/mingw/mingw",
> but cloning that yields an empty repo.  WTH?  Is the project actively
> trying to hide itself in some dark internet corner?

I have quite recent sources here, and the feature is there.  Thanks.