gcc structures

gcc structures

[JimJoyce]

Is this a bug?
I'm coding in plain 'C'.
I am trying to read and write GIS shapefiles.
They start with a header of 9 ints and 8 doubles.

So I used a structure eg:
struct hdr { int g1[9]; double g2[8] }; struc;
 fread ( struc, 100, 1, fp1);
It was screwing up my doubles.
-- 
When, eventually, I experimented:
sizeof(g1); sizeof(g2), sizeof (struc).
I got 36, 64, 104.
Note 104, not 100.

Where is gcc placing the redundant 4 bytes?
Is 'struct' insisting on a doubleword boundary?

NB I got round the problem by using 2 fread()s
fread(struc.g1,36,1,fp1);
fread(struc.g2,64,1,fp1);



--
View this message in context: http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Re: gcc structures

[Jonathan Wakely]

On 8 September 2013 11:10, JimJoycewrote:
> Is this a bug?

No.

> I'm coding in plain 'C'.
> I am trying to read and write GIS shapefiles.
> They start with a header of 9 ints and 8 doubles.
>
> So I used a structure eg:
> struct hdr { int g1[9]; double g2[8] }; struc;
>  fread ( struc, 100, 1, fp1);
> It was screwing up my doubles.
> --
> When, eventually, I experimented:
> sizeof(g1); sizeof(g2), sizeof (struc).
> I got 36, 64, 104.
> Note 104, not 100.
>
> Where is gcc placing the redundant 4 bytes?

Between the two arrays. The array of doubles is 8-byte aligned,
presumably because that's what your architecture requires.

> Is 'struct' insisting on a doubleword boundary?

For the double array, yes.  The C standard doesn't say struct members
must be adjacent, padding is allowed.

> NB I got round the problem by using 2 fread()s
> fread(struc.g1,36,1,fp1);
> fread(struc.g2,64,1,fp1);

That's the right thing to do.

You could also try this:

struct __attribute__((__packed__)) hdr { int g1[9]; double g2[8]; };

Although the compiler adds the padding for good reasons so it's best
not to force it to use a different layout unless you really need to.

Re: gcc structures

[JimJoyce]

Thanks, Jonathan, for your speedy reply.

However, I'm surprised, That 'C' can pad structures as it sees fit.
I thought the point and value of user-defined structures was to suit user's
needs.
not the whim of the compiler..

I think you're telling me that   '__attribute__((__packed__))'   is what I
need to do  to force the compiler to do things my way?

I learned my coding in the old days, using IBM 360 Assembler.
We programmers told the system how we wanted things.
Happy Days !

Thanks, JJ


On 8 September 2013 13:30, Jonathan Wakely-4 [via gcc] <
ml-node+s1065356n966604h15@n5.nabble.com> wrote:

> On 8 September 2013 11:10, JimJoycewrote:
> > Is this a bug?
>
> No.
>
> > I'm coding in plain 'C'.
> > I am trying to read and write GIS shapefiles.
> > They start with a header of 9 ints and 8 doubles.
> >
> > So I used a structure eg:
> > struct hdr { int g1[9]; double g2[8] }; struc;
> >  fread ( struc, 100, 1, fp1);
> > It was screwing up my doubles.
> > --
> > When, eventually, I experimented:
> > sizeof(g1); sizeof(g2), sizeof (struc).
> > I got 36, 64, 104.
> > Note 104, not 100.
> >
> > Where is gcc placing the redundant 4 bytes?
>
> Between the two arrays. The array of doubles is 8-byte aligned,
> presumably because that's what your architecture requires.
>
> > Is 'struct' insisting on a doubleword boundary?
>
> For the double array, yes.  The C standard doesn't say struct members
> must be adjacent, padding is allowed.
>
> > NB I got round the problem by using 2 fread()s
> > fread(struc.g1,36,1,fp1);
> > fread(struc.g2,64,1,fp1);
>
> That's the right thing to do.
>
> You could also try this:
>
> struct __attribute__((__packed__)) hdr { int g1[9]; double g2[8]; };
>
> Although the compiler adds the padding for good reasons so it's best
> not to force it to use a different layout unless you really need to.
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
> http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966604.html
>  To unsubscribe from gcc structures, click here<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=966595&code=amltQGppbWpveWNlLmNvLnVrfDk2NjU5NXwtOTM4MDcwNDA4>
> .
> NAML<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>



-- 


JimJoyce
T: 0(044) 1280 813 899
E: jim@jimjoyce.co.uk




--
View this message in context: http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966606.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Fwd: gcc structures

[JimJoyce]

Thanks, again.  JJ.


---------- Forwarded message ----------
From: Jim Joyce <jim@jimjoyce.co.uk>
Date: 8 September 2013 14:21
Subject: Re: gcc structures
To: "Jonathan Wakely-4 [via gcc]" <ml-node+s1065356n966604h15@n5.nabble.com>


Thanks, Jonathan, for your speedy reply.

However, I'm surprised, That 'C' can pad structures as it sees fit.
I thought the point and value of user-defined structures was to suit user's
needs.
not the whim of the compiler..

I think you're telling me that   '__attribute__((__packed__))'   is what I
need to do  to force the compiler to do things my way?

I learned my coding in the old days, using IBM 360 Assembler.
We programmers told the system how we wanted things.
Happy Days !

Thanks, JJ


On 8 September 2013 13:30, Jonathan Wakely-4 [via gcc] <
ml-node+s1065356n966604h15@n5.nabble.com> wrote:

> On 8 September 2013 11:10, JimJoycewrote:
> > Is this a bug?
>
> No.
>
> > I'm coding in plain 'C'.
> > I am trying to read and write GIS shapefiles.
> > They start with a header of 9 ints and 8 doubles.
> >
> > So I used a structure eg:
> > struct hdr { int g1[9]; double g2[8] }; struc;
> >  fread ( struc, 100, 1, fp1);
> > It was screwing up my doubles.
> > --
> > When, eventually, I experimented:
> > sizeof(g1); sizeof(g2), sizeof (struc).
> > I got 36, 64, 104.
> > Note 104, not 100.
> >
> > Where is gcc placing the redundant 4 bytes?
>
> Between the two arrays. The array of doubles is 8-byte aligned,
> presumably because that's what your architecture requires.
>
> > Is 'struct' insisting on a doubleword boundary?
>
> For the double array, yes.  The C standard doesn't say struct members
> must be adjacent, padding is allowed.
>
> > NB I got round the problem by using 2 fread()s
> > fread(struc.g1,36,1,fp1);
> > fread(struc.g2,64,1,fp1);
>
> That's the right thing to do.
>
> You could also try this:
>
> struct __attribute__((__packed__)) hdr { int g1[9]; double g2[8]; };
>
> Although the compiler adds the padding for good reasons so it's best
> not to force it to use a different layout unless you really need to.
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
> http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966604.html
>  To unsubscribe from gcc structures, click here<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=966595&code=amltQGppbWpveWNlLmNvLnVrfDk2NjU5NXwtOTM4MDcwNDA4>
> .
> NAML<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>



-- 


JimJoyce
T: 0(044) 1280 813 899
E: jim@jimjoyce.co.uk



My question  'Are Structures doublewoord aligned?' was not about the second
half of the structure, but the beginning: an array of 9 ints.
It appears, having decided to place them on a doubleword boundary, it then
had to pad after the 9 ints to get back to an 8-byte boundary.
Was it pure mischance that the structure happened to start on a doubleword
that the extra int was needed. Had it started 4 bytes later, there would be
no padding? Or do structures always start on a doubleword?

Thanks, again,
JJ
-- 


JimJoyce
T: 0(044) 1280 813 899
E: jim@jimjoyce.co.uk




--
View this message in context: http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966610.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Re: gcc structures

[Jonathan Wakely]

On 8 September 2013 14:22, JimJoyce wrote:
> Thanks, Jonathan, for your speedy reply.
>
> However, I'm surprised, That 'C' can pad structures as it sees fit.
> I thought the point and value of user-defined structures was to suit user's
> needs.
> not the whim of the compiler..

It's not the whim of the compiler, it's due to the hardware. See
http://c-faq.com/struct/padding.html and the links on that page.

> I think you're telling me that   '__attribute__((__packed__))'   is what I
> need to do  to force the compiler to do things my way?

Yes, but that's not portable and might slow the code down if the
hardware has to perform unaligned accesses to doubles.

Re: gcc structures

[Jonathan Wakely]

On 8 September 2013 14:42, JimJoyce wrote:
>
> My question  'Are Structures doublewoord aligned?' was not about the second
> half of the structure, but the beginning: an array of 9 ints.
> It appears, having decided to place them on a doubleword boundary, it then
> had to pad after the 9 ints to get back to an 8-byte boundary.

Yes, there's a "hole" in the middle of the struct.

> Was it pure mischance that the structure happened to start on a doubleword
> that the extra int was needed. Had it started 4 bytes later, there would be
> no padding?

If you think about it that question doesn't make sense. The layout of
a struct is always the same, irrespective of where an particular
instance of that struct happens to be positioned in memory.  A
struct's definition does not "start" anywhere in memory, only an
instance of the struct has an address.

> Or do structures always start on a doubleword?

It depends on the types in the struct and the ABI of the target platform.

There's plenty of information about this on the web, e.g.
http://en.wikipedia.org/wiki/Data_structure_alignment#Typical_alignment_of_C_structs_on_x86

Re: gcc structures

[JimJoyce]

Thanks, Jonathan, for your patience, and detailed answer.
I'll need to take some time to digest he wikipedia article.

I was thinking of structure instances, not definitions, when questioning
their positioning and padding.

I can see now that an instance of a structure of 9 ints followed by 8
doubles
would need to be positioned carefully to avoid that dreaded hole.

Thanks, but I may need to come back to you after studying that article.

JJ


On 8 September 2013 15:33, Jonathan Wakely-4 [via gcc] <
ml-node+s1065356n966628h59@n5.nabble.com> wrote:

> On 8 September 2013 14:42, JimJoyce wrote:
> >
> > My question  'Are Structures doublewoord aligned?' was not about the
> second
> > half of the structure, but the beginning: an array of 9 ints.
> > It appears, having decided to place them on a doubleword boundary, it
> then
> > had to pad after the 9 ints to get back to an 8-byte boundary.
>
> Yes, there's a "hole" in the middle of the struct.
>
> > Was it pure mischance that the structure happened to start on a
> doubleword
> > that the extra int was needed. Had it started 4 bytes later, there would
> be
> > no padding?
>
> If you think about it that question doesn't make sense. The layout of
> a struct is always the same, irrespective of where an particular
> instance of that struct happens to be positioned in memory.  A
> struct's definition does not "start" anywhere in memory, only an
> instance of the struct has an address.
>
> > Or do structures always start on a doubleword?
>
> It depends on the types in the struct and the ABI of the target platform.
>
> There's plenty of information about this on the web, e.g.
>
> http://en.wikipedia.org/wiki/Data_structure_alignment#Typical_alignment_of_C_structs_on_x86
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
> http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966628.html
>  To unsubscribe from gcc structures, click here<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=966595&code=amltQGppbWpveWNlLmNvLnVrfDk2NjU5NXwtOTM4MDcwNDA4>
> .
> NAML<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>



-- 


JimJoyce
T: 0(044) 1280 813 899
E: jim@jimjoyce.co.uk




--
View this message in context: http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966657.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Re: gcc structures

[Max S.]

How will you use this struct once you have it aligned the way you want?

If you use it as a tx/rx buffer, be aware that doubles aren't portable.
Let alone that ints could be different size on different machines.
Even if your just writing these structs to file. The resulting file will
not be portable across architectures.

//bar is packed; slower to access.
struct special_packed_machine_dependent_struct bar = ...;

//result is binary compatible only with similar machines.
write(fd, &bar, sizeof(bar));

If the task is small It may be better to fiddle the bits/bytes yourself.

uint8_t buff[HEADE_SIZE];

get_data_from_someplace(buff);

//bar is not aligned. therefore possibly faster to access.
struct special_machine_dependent_struct bar;

//construct MEMBER preserving byte order.
bar.member = (buff[U16_MEMBER_OFFSET+1]<<8)|(buff[U16_MEMBER_OFFSET]);

I don't know what GIS shapefiles are so... The above may be pure lunacy.

MS.

Re: gcc structures

[David Brown]

On 08/09/13 15:22, JimJoyce wrote:
> Thanks, Jonathan, for your speedy reply.
> 
> However, I'm surprised, That 'C' can pad structures as it sees fit.
> I thought the point and value of user-defined structures was to suit user's
> needs.
> not the whim of the compiler..

The C compiler is allowed some leeway.  I'm not even sure if it is
required to keep the struct elements in the same order (there was a gcc
option to allow it to change the order in certain circumstances, but I
gather it has now been removed since it worked badly with LTO) - it just
has to produce code that /acts/ as though the order is as given.

In particular, the compiler will normally pad elements in a struct as
necessary to fit the alignment requirements of your architecture.  On
some architectures, incorrect alignment will mean the program does not
work - it will either work with incorrect data, or trip a processor
exception.  On others, incorrect alignment will merely mean the code
runs slowly.

When you are concerned about the exact format of your structs, I
strongly recommend using the "-Wpadded" switch so that the compiler will
inform you of any added padding bytes.  Then you can adapt your struct
to fit - possibly by adding explicit "padding" entries to make
everything fit correctly.

David


> 
> I think you're telling me that   '__attribute__((__packed__))'   is what I
> need to do  to force the compiler to do things my way?
> 
> I learned my coding in the old days, using IBM 360 Assembler.
> We programmers told the system how we wanted things.
> Happy Days !
> 
> Thanks, JJ
> 
> 
> On 8 September 2013 13:30, Jonathan Wakely-4 [via gcc] <
> ml-node+s1065356n966604h15@n5.nabble.com> wrote:
> 
>> On 8 September 2013 11:10, JimJoycewrote:
>>> Is this a bug?
>>
>> No.
>>
>>> I'm coding in plain 'C'.
>>> I am trying to read and write GIS shapefiles.
>>> They start with a header of 9 ints and 8 doubles.
>>>
>>> So I used a structure eg:
>>> struct hdr { int g1[9]; double g2[8] }; struc;
>>>  fread ( struc, 100, 1, fp1);
>>> It was screwing up my doubles.
>>> --
>>> When, eventually, I experimented:
>>> sizeof(g1); sizeof(g2), sizeof (struc).
>>> I got 36, 64, 104.
>>> Note 104, not 100.
>>>
>>> Where is gcc placing the redundant 4 bytes?
>>
>> Between the two arrays. The array of doubles is 8-byte aligned,
>> presumably because that's what your architecture requires.
>>
>>> Is 'struct' insisting on a doubleword boundary?
>>
>> For the double array, yes.  The C standard doesn't say struct members
>> must be adjacent, padding is allowed.
>>
>>> NB I got round the problem by using 2 fread()s
>>> fread(struc.g1,36,1,fp1);
>>> fread(struc.g2,64,1,fp1);
>>
>> That's the right thing to do.
>>
>> You could also try this:
>>
>> struct __attribute__((__packed__)) hdr { int g1[9]; double g2[8]; };
>>
>> Although the compiler adds the padding for good reasons so it's best
>> not to force it to use a different layout unless you really need to.
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the discussion
>> below:
>> http://gcc.1065356.n5.nabble.com/gcc-structures-tp966595p966604.html
>>  To unsubscribe from gcc structures, click here<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=966595&code=amltQGppbWpveWNlLmNvLnVrfDk2NjU5NXwtOTM4MDcwNDA4>
>> .
>> NAML<http://gcc.1065356.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
> 
> 
> 

Re: gcc structures

[Andrew Haley]

On 09/09/2013 09:57 AM, David Brown wrote:
> I'm not even sure if it is
> required to keep the struct elements in the same order

It is.  6.7.2.1:

    Each non-bit-field member of a structure or union object is
    aligned in an implementation-defined manner appropriate to its
    type. Within a structure object, the non-bit-field members and the
    units in which bit-fields reside have addresses that increase in
    the order in which they are declared.

Andrew.

Re: gcc structures

[Donald R Laster Jr]

  Here is some more information on structure alignments and a little bit of history based upon my experience with 8 to 10 different vendor/versions of "C" and C++ compilers over the years.

  The alignment issue is one reason why when I am creating structures in "C" and C++ I pay close attention to the sizes of the variables in the structure - especially arrays and other structures.  If the data is going to be written to files or being transmitted between systems it becomes even more important.  CORBA and other transport mechanism will hide much of these issue from the application programmer in some cases.

  What I have found is all of the compilers will generally place variables on the required or expected alignment based upon the variable type (int, float, double) by default and hardware machine type.  If the structure remains local to the system in memory you would normally never notice any issues.  If you start moving the data around, into files or to different systems, you can run into problems.  If you were to use "sizeof(struct name)" instead of 100, you would probably not have noticed the issue since the size of the data being read and written would have included the alignment fullword (4 bytes) between g1 and g2.  Only if you looked at the size of the file would you have had a question.

  In some cases I have manually placed alignment variables to insure the alignment placement is obvious.  It becomes very important when dealing with low level data accesses from files or across a network.  I have seen code written that transmits a structure across a network to a system with a different alignment requirement and the results on the other end are not what is expected.  In some cases the size of the data transmitted is less or more than the other end expected.  The person writing the code counted the size of the individual variables and arrays instead of the actual size of the structure.  Debugging the problem initially was not easy for various reasons.

  Consider this structure 

      struct  words {		/*	A		B  */
        signed long int   int4;	/*	0		0  */
        double            flt8;	/*	4		8  */
      };			/*size 12	  size 16  */

It may be 12 bytes long or 16 bytes long depending upon the architectures I have used.  On older 32 bit hardware (IBMs, DEC, CCUR (PE), Data General, Gould - think 1980s/early 1990s) platforms the offsets and size are from column A (what you expected I believe).  The requirement was for 4 byte alignment of 32 bit and 64 bit values.  While on newer versions of the compilers and on 64 bit hardware (later IBMS, later CCUR (PE), Sparc v8/v9, etc) the offsets and size are from column B.  Variables that are 8 bytes in size are generally placed on 8 byte boundaries by default today.  It makes it simpler to write compilers and not create problems when moving code to different platforms.  

  The newer 64 bit systems expects variables that are 64 bits (8 bytes) in size (long long's, doubles) to be on 8 byte boundaries or alignment exceptions occur.  Thus the compiler places variables on the natural alignment based upon the size of the variable.  On Intel chips these data types can still be accessed on different alignments (to my knowledge) but the performance drops significantly since the chips have to do extra work to get the data to and from memory and the compiler has to generate more machine code to do the work as well.  

  Older compilers for Intel architectures did not care since the memory accesses were generally byte oriented and it did not matter if it was char (1), halfword (2), fullword (4) or doubleword (8) aligned.  This goes back to the 8086, 80286 and Z80 days.  

  Another thing you need to be aware of is the "Little-Endian" and "Big-Endian" issue.  Especially, if you are moving data across platforms -  such as Intel/AMD to SPARC/IBM or vice-versa.

  Hope this help some.

   Don

David Brown wrote:
> On 08/09/13 15:22, JimJoyce wrote:
>> Thanks, Jonathan, for your speedy reply.
>>
>> However, I'm surprised, That 'C' can pad structures as it sees fit.
>> I thought the point and value of user-defined structures was to suit user's
>> needs.
>> not the whim of the compiler..
> 
> The C compiler is allowed some leeway.  I'm not even sure if it is
> required to keep the struct elements in the same order (there was a gcc
> option to allow it to change the order in certain circumstances, but I
> gather it has now been removed since it worked badly with LTO) - it just
> has to produce code that /acts/ as though the order is as given.
> 
> In particular, the compiler will normally pad elements in a struct as
> necessary to fit the alignment requirements of your architecture.  On
> some architectures, incorrect alignment will mean the program does not
> work - it will either work with incorrect data, or trip a processor
> exception.  On others, incorrect alignment will merely mean the code
> runs slowly.
> 
> When you are concerned about the exact format of your structs, I
> strongly recommend using the "-Wpadded" switch so that the compiler will
> inform you of any added padding bytes.  Then you can adapt your struct
> to fit - possibly by adding explicit "padding" entries to make
> everything fit correctly.
> 
> David
> 
>