coff section flags: STYP_COPY, in particular, for TI COFF files

coff section flags: STYP_COPY, in particular, for TI COFF files

[John E Hein]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 979 bytes --]

I am looking into supporting the TMS320C6x
(c6x-tools.sourceforge.net), and I've found that the COFF files
generated by TI's compiler (Code Composer Studio) include some
sections flagged by STYP_COPY.

This is not handled in bfd/coffcode.h

Sections flagged by STYP_COPY are to be relocated and loaded, but not
allocated.  For instance, there's a .vers section that has section
flags 0x50 (STYP_COPY | STYP_DATA).  binutils marks these with
SEC_LOAD & SEC_ALLOC (loaded & allocated).

In recent TI tools (CCS 2.20), there's another section ($BRID) which
has the same problem.

Note that the c54x support that is now included in binutils has the
same flaw.

There are also problems with STYP_DSECT (not handled by binutils in
sec_to_styp_flags()) and also with STYP_PAD (should be loaded, but not
marked by binutils with SEC_LOAD).  But I haven't seen a COFF file
generated by the TI tools that have the STYP_DSECT nor STYP_PAD flags
set (yet).

Attached is a patch.  Comments?



[-- Attachment #2: coffcode.h STYP_COPY, et. al., patch --]
[-- Type: text/plain, Size: 753 bytes --]

--- coffcode.h.orig	Fri Apr  4 17:41:35 2003
+++ coffcode.h	Fri Apr  4 17:43:41 2003
@@ -644,8 +644,10 @@
       sec_flags |= SEC_DEBUGGING;
 #endif
     }
+  else if ((styp_flags & STYP_PAD) && (sec_flags & SEC_NEVER_LOAD))
+	sec_flags = 0;
   else if (styp_flags & STYP_PAD)
-    sec_flags = 0;
+	sec_flags = SEC_LOAD;
   else if (strcmp (name, _TEXT) == 0)
     {
       if (sec_flags & SEC_NEVER_LOAD)
@@ -702,6 +704,9 @@
   if (styp_flags & STYP_OTHER_LOAD)
     sec_flags = (SEC_LOAD | SEC_ALLOC);
 #endif /* STYP_SDATA */
+
+  if (styp_flags & (STYP_COPY | STYP_DSECT))
+	sec_flags &= ~SEC_ALLOC);
 
 #if defined (COFF_LONG_SECTION_NAMES) && defined (COFF_SUPPORT_GNU_LINKONCE)
   /* As a GNU extension, if the name begins with .gnu.linkonce, we

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[Ian Lance Taylor]

John E Hein <jhein@timing.com> writes:

> Sections flagged by STYP_COPY are to be relocated and loaded, but not
> allocated.  For instance, there's a .vers section that has section
> flags 0x50 (STYP_COPY | STYP_DATA).  binutils marks these with
> SEC_LOAD & SEC_ALLOC (loaded & allocated).

How can something be loaded but not allocated?

In BFD lingo, loaded means that the section should be loaded into
memory when the program is run.  Allocated means that a portion of
memory should be set aside for the section at link time.  It's
possible to have a section which is allocated but not loaded: the .bss
section is one--there is no need to load it because the contents are
known to be zero.  But if a section is loaded but not allocated, then
where are you supposed to load it?

Ian

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[Michael S. Zick]

On Saturday 05 April 2003 00:48, Ian Lance Taylor wrote:
> John E Hein <jhein@timing.com> writes:
> But if a section is loaded but not allocated, then
> where are you supposed to load it?

Overlays? Load to previously allocated area.
Crash & Burn? Overwrite kernel space?
Mike

>
> Ian

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[John E Hein]

Ian Lance Taylor wrote at 22:48 -0800 on Apr  4:
 > John E Hein <jhein@timing.com> writes:
 > 
 > > Sections flagged by STYP_COPY are to be relocated and loaded, but
 > > not allocated.  For instance, there's a .vers section that has
 > > section flags 0x50 (STYP_COPY | STYP_DATA).  binutils marks these
 > > with SEC_LOAD & SEC_ALLOC (loaded & allocated).
 > 
 > How can something be loaded but not allocated?

I believe...
- allocated means space is reserved in the target memory at "run" time
- loaded means loaded into the coff image itself, not necessarily
  put in the target memory

see comments in include/coff/internal.h, look for STYP_* and the
comments about which sections should be relocated, allocate &/or
loaded or not.

From the coff description in the TI docs
(http://www-s.ti.com/sc/psheets/spru186l/spru186l.pdf, p. A-7)...

'The term "loaded" means that the raw data for this section appears in
the object file.'


 > In BFD lingo, loaded means that the section should be loaded into
 > memory when the program is run. Allocated means that a portion of
 > memory should be set aside for the section at link time.

Then the STYP_* comments in binutils/include/coff/internal.h are in
conflict.  In the binutils include/coffcode.h, it certainly sets
SEC_ALLOC and not SEC_LOAD in some cases (STYP_BSS, for instance).  If
'allocated' is just a link time directive, then how does the OS know
at run time what to do with it?


 > It's possible to have a section which is allocated but not loaded:
 > the .bss section is one--there is no need to load it because the
 > contents are known to be zero.  But if a section is loaded but not
 > allocated, then where are you supposed to load it?

This sounds opposite of what you just described.  I would expect .bss
to be initialized in memory with all zeroes at run time ('loaded' in
your explanation of the bfd lingo).  Maybe that's what you mean by
allocated (except that it's also set to 0s at some point in addition
to just being allocated).

FYI, the coff STYP_PAD section is "loaded, not allocated"
(internal.h), for instance, as is STYP_COPY.

It's entirely possible that someone (TI?) has this all backwards?
Or I'm just missing a piece in my mind's eye somewhere.


In any case, the main bit I'm trying to work out is this... the image
produced by the TI tools that has a section that should not be put
into memory at run time has 0x50 section flags (STYP_DATA |
STYP_COPY).  According to the binutils comments in coff/internal.h,
STYP_COPY means loaded, not allocated.  If loaded means what you
describe, as opposed to what the TI doc describes, then something is
wrong.

And the binutils coff code currently does nothing at all if STYP_COPY
is set.

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[Ian Lance Taylor]

John E Hein <jhein@timing.com> writes:

> Ian Lance Taylor wrote at 22:48 -0800 on Apr  4:
>  > John E Hein <jhein@timing.com> writes:
>  > 
>  > > Sections flagged by STYP_COPY are to be relocated and loaded, but
>  > > not allocated.  For instance, there's a .vers section that has
>  > > section flags 0x50 (STYP_COPY | STYP_DATA).  binutils marks these
>  > > with SEC_LOAD & SEC_ALLOC (loaded & allocated).
>  > 
>  > How can something be loaded but not allocated?
> 
> I believe...
> - allocated means space is reserved in the target memory at "run" time
> - loaded means loaded into the coff image itself, not necessarily
>   put in the target memory

In BFD terminology, that corresponds to SEC_HAS_CONTENTS.

> see comments in include/coff/internal.h, look for STYP_* and the
> comments about which sections should be relocated, allocate &/or
> loaded or not.

Those comments are very old, and may or may not correspond to the BFD
section flags.

>  > In BFD lingo, loaded means that the section should be loaded into
>  > memory when the program is run. Allocated means that a portion of
>  > memory should be set aside for the section at link time.
> 
> Then the STYP_* comments in binutils/include/coff/internal.h are in
> conflict.  In the binutils include/coffcode.h, it certainly sets
> SEC_ALLOC and not SEC_LOAD in some cases (STYP_BSS, for instance).  If
> 'allocated' is just a link time directive, then how does the OS know
> at run time what to do with it?

``Allocated'' is not just a link time directive.  The OS must also
know how to handle such a section, of course, once the linker has
decided where it is to go.

>  > It's possible to have a section which is allocated but not
>  > loaded: the .bss section is one--there is no need to load it
>  > because the contents are known to be zero.  But if a section is
>  > loaded but not allocated, then where are you supposed to load it?
> 
> This sounds opposite of what you just described.  I would expect .bss
> to be initialized in memory with all zeroes at run time ('loaded' in
> your explanation of the bfd lingo).  Maybe that's what you mean by
> allocated (except that it's also set to 0s at some point in addition
> to just being allocated).

There is a misunderstanding here.  ``Loaded'' means that the section
contents are copied from the executable file into memory at execution
time.  This is not true of a .bss section--there are no section
contents to copy.  ``Allocated'' means that space is set aside in the
execution time memory map; if the section is not also loaded, then
typically that space is zeroed (often implicitly via demand paging) at
execution time.  Ordinary text and data sections are both ``loaded''
and ``allocated.''  BSS sections are ``allocated'' but not ``loaded.''

> FYI, the coff STYP_PAD section is "loaded, not allocated"
> (internal.h), for instance, as is STYP_COPY.

I don't know how STYP_PAD and STYP_COPY sections are to be handled, so
I don't know how they should be translated into BFD section flags.

> It's entirely possible that someone (TI?) has this all backwards?
> Or I'm just missing a piece in my mind's eye somewhere.

It does sound like the terminology is being used in some sort of
reverse sense for some reason.

> In any case, the main bit I'm trying to work out is this... the image
> produced by the TI tools that has a section that should not be put
> into memory at run time has 0x50 section flags (STYP_DATA |
> STYP_COPY).  According to the binutils comments in coff/internal.h,
> STYP_COPY means loaded, not allocated.  If loaded means what you
> describe, as opposed to what the TI doc describes, then something is
> wrong.
> 
> And the binutils coff code currently does nothing at all if STYP_COPY
> is set.

Can you describe in more detail what is supposed to happen with a
STYP_COPY section?  What would be an example use of such a section?

Ignore the comments in coff/internal.h.  They may be accurate, or they
may not, and in any case the version of STYP_COPY they describe may
not be the one expected by the TI tools.

Ian

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[John E Hein]

Ian Lance Taylor wrote at 13:29 -0800 on Apr  5:
 > John E Hein <jhein@timing.com> writes:
 > > I believe...
 > > - allocated means space is reserved in the target memory at "run" time
 > > - loaded means loaded into the coff image itself, not necessarily
 > >   put in the target memory
 > 
 > In BFD terminology, that corresponds to SEC_HAS_CONTENTS.

In bfd, SEC_HAS_CONTENTS is set if the section size != 0 or if
SEC_LOAD is not set (which is true in the case of certain STYP_*
flags - see coffcode.h's styp_to_sec_flags()).


 > > see comments in include/coff/internal.h, look for STYP_* and the
 > > comments about which sections should be relocated, allocate &/or
 > > loaded or not.
 > 
 > Those comments are very old, and may or may not correspond to the BFD
 > section flags.

Maybe so, but I assumed that's why there's a styp_to_sec_flags
function in coffcode.h - to translate between the coff view and the bfd
view.


 > >  > In BFD lingo, loaded means that the section should be loaded into
 > >  > memory when the program is run. Allocated means that a portion of
 > >  > memory should be set aside for the section at link time.
 > > 
 > > Then the STYP_* comments in binutils/include/coff/internal.h are in
 > > conflict.  In the binutils include/coffcode.h, it certainly sets
 > > SEC_ALLOC and not SEC_LOAD in some cases (STYP_BSS, for instance).  If
 > > 'allocated' is just a link time directive, then how does the OS know
 > > at run time what to do with it?
 > 
 > ``Allocated'' is not just a link time directive.  The OS must also
 > know how to handle such a section, of course, once the linker has
 > decided where it is to go.

Okay... you had mentioned in your description above that it was a
link time thing, and I misunderstood what you were getting at.


 > >  > It's possible to have a section which is allocated but not
 > >  > loaded: the .bss section is one--there is no need to load it
 > >  > because the contents are known to be zero.  But if a section is
 > >  > loaded but not allocated, then where are you supposed to load it?
 > > 
 > > This sounds opposite of what you just described.  I would expect .bss
 > > to be initialized in memory with all zeroes at run time ('loaded' in
 > > your explanation of the bfd lingo).  Maybe that's what you mean by
 > > allocated (except that it's also set to 0s at some point in addition
 > > to just being allocated).
 > 
 > There is a misunderstanding here.  ``Loaded'' means that the section
 > contents are copied from the executable file into memory at execution
 > time.

That was my original notion of the term, but as I said, the TI
explanation of the coff section flags, says: 'The term "loaded" means
that the raw data for this section appears in the object file.'


 > This is not true of a .bss section--there are no section
 > contents to copy.  ``Allocated'' means that space is set aside in the
 > execution time memory map; if the section is not also loaded, then
 > typically that space is zeroed (often implicitly via demand paging) at
 > execution time.  Ordinary text and data sections are both ``loaded''
 > and ``allocated.''  BSS sections are ``allocated'' but not ``loaded.''

Is there a knob to say allocated and zeroed?  Or is that implicit in
all allocated pieces?  You said "typically" zeroed... are there actual
known cases where it's not?  I imagine that would break some apps.
But this is a sidetrack anyway...


 > > FYI, the coff STYP_PAD section is "loaded, not allocated"
 > > (internal.h), for instance, as is STYP_COPY.
 > 
 > I don't know how STYP_PAD and STYP_COPY sections are to be handled, so
 > I don't know how they should be translated into BFD section flags.

I can only go on the comments that I see in internal.h (and I've seen
elsewhere) and the comments in the TI doc (which are very similar
except perhaps for what "loaded" means).

internal.h
----------
STYP_PAD   "padding": not allocated, not relocated, loaded

STYP_COPY  "copy": for decision function used by field update;
           not allocated, not relocated, loaded;
           reloc & lineno entries processed normally

spru186l.pdf & spru102f.pdf (TI docs for c6x & c54x processors, c54x
support which is now part of binutils)
----------------
STYP_PAD   loaded, not allocated, not relocated

STYP_COPY  loaded, not allocated, relocated; relocate & line number
            entries processed normally

The only diff is relocated vs. not relocated for STYP_COPY, but I don't
care about that for what I am trying to figure out.


 > > In any case, the main bit I'm trying to work out is this... the image
 > > produced by the TI tools that has a section that should not be put
 > > into memory at run time has 0x50 section flags (STYP_DATA |
 > > STYP_COPY).  According to the binutils comments in coff/internal.h,
 > > STYP_COPY means loaded, not allocated.  If loaded means what you
 > > describe, as opposed to what the TI doc describes, then something is
 > > wrong.
 > > 
 > > And the binutils coff code currently does nothing at all if STYP_COPY
 > > is set.
 > 
 > Can you describe in more detail what is supposed to happen with a
 > STYP_COPY section?  What would be an example use of such a section?

According to TI support...

"the documentation for copy section says these section will get loaded
but not allocated in the memory"

Again, remember their doc says:
'The term "loaded" means that the raw data for this section appears in
the object file.'

BTW, the BFD section flags for the STYP_COPY | STYP_DATA section are
0x243 (CONTENTS, ALLOC, LOAD, DATA).

But just plain STYP_DATA sections (like .const) also get 0x243.


 > Ignore the comments in coff/internal.h.  They may be accurate, or they
 > may not, and in any case the version of STYP_COPY they describe may
 > not be the one expected by the TI tools.

Okay, so what are the correct docs to look at?  Any coff gurus care to
chime in (this is supposedly coff "version 2", btw)?  I've seen
snippets in various google searches that seem to have the same text as
the comments in binutils internal.h; many search hits say very little
about section types beyond text, data & bss.

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[Ian Lance Taylor]

John E Hein <jhein@timing.com> writes:

>  > Can you describe in more detail what is supposed to happen with a
>  > STYP_COPY section?  What would be an example use of such a section?
> 
> According to TI support...
> 
> "the documentation for copy section says these section will get loaded
> but not allocated in the memory"
> 
> Again, remember their doc says:
> 'The term "loaded" means that the raw data for this section appears in
> the object file.'
> 
> BTW, the BFD section flags for the STYP_COPY | STYP_DATA section are
> 0x243 (CONTENTS, ALLOC, LOAD, DATA).
> 
> But just plain STYP_DATA sections (like .const) also get 0x243.

Well, given their definition of ``loaded,'' it sounds to me like an
STYP_COPY section is one which has contents, but the contents are not
copied into memory at execution time.  This is true, for example, of a
section which holds debugging information.  In BFD terms, I think such
a section would be SEC_HAS_CONTENTS, but no other flags (except maybe
SEC_RELOC).

The only way to know whether this makes any sense is to examine the
actual behaviour of the TI tools when given a STYP_COPY section.  For
example: how is an STYP_COPY section created in the first place?  What
happens to an STYP_COPY section at link time?  At run time?

>  > Ignore the comments in coff/internal.h.  They may be accurate, or they
>  > may not, and in any case the version of STYP_COPY they describe may
>  > not be the one expected by the TI tools.
> 
> Okay, so what are the correct docs to look at?  Any coff gurus care to
> chime in (this is supposedly coff "version 2", btw)?  I've seen
> snippets in various google searches that seem to have the same text as
> the comments in binutils internal.h; many search hits say very little
> about section types beyond text, data & bss.

One of the reasons why people should avoid COFF is that it was never
properly documented.  The AT&T docs said nothing useful.  There was an
O'Reilly book a while back, now out of print, which is more or less
``everything you wanted to know about COFF except what you really need
to know.''

I checked the O'Reilly book, and it does not mention STYP_COPY.  My
SVR3 docs are sufficiently buried that I don't really feel like
searching for them.  I would be surprised if they mention STYP_COPY.

Ian

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[John E Hein]

Ian Lance Taylor wrote at 19:55 -0700 on Apr  6:
 > Well, given their definition of ``loaded,'' it sounds to me like an
 > STYP_COPY section is one which has contents, but the contents are not
 > copied into memory at execution time.  This is true, for example, of a
 > section which holds debugging information.  In BFD terms, I think such
 > a section would be SEC_HAS_CONTENTS, but no other flags (except maybe
 > SEC_RELOC).

Right, that's what I was trying to do with the original patch to
coffcode.h (where I tried to handle the previously unhandled STYP_COPY
in a way consistent with the comments in internal.h).  Unfortunately,
the general COFF code may not be the place if TI's COFF treatment is
sufficiently abnormal.

But I did try to match the comments in internal.h - if those are
wrong, I suppose they should be fixed - or words added to indicate
that different COFF implementation may differ with their intended use
of STYP_* flags.

Instead of modifying the general COFF handling in binutils, I may
rather want to override that behavior for TI COFF.


 > The only way to know whether this makes any sense is to examine the
 > actual behaviour of the TI tools when given a STYP_COPY section.  For
 > example: how is an STYP_COPY section created in the first place?  What
 > happens to an STYP_COPY section at link time?  At run time?

I have asked TI support for a more detailed explanation regarding how
their tools behave for various section types.  We'll see what they
have to say.


 > One of the reasons why people should avoid COFF is that it was never
 > properly documented.  The AT&T docs said nothing useful.  There was an
 > O'Reilly book a while back, now out of print, which is more or less
 > ``everything you wanted to know about COFF except what you really need
 > to know.''

So, I'm finding out.


 > I checked the O'Reilly book, and it does not mention STYP_COPY.  My
 > SVR3 docs are sufficiently buried that I don't really feel like
 > searching for them.  I would be surprised if they mention STYP_COPY.

Someone else here had the O'Reilly COFF book, too (1988).  As you say,
it wasn't much help.

Does anyone know who was the original porter(s) of C54x to binutils?
Maybe they can point to something that clears it up.

Re: coff section flags: STYP_COPY, in particular, for TI COFF files

[John E Hein]

John E Hein wrote at 10:54 -0600 on Apr  7:
 > I have asked TI support for a more detailed explanation regarding how
 > their tools behave for various section types.  We'll see what they
 > have to say.

TI has confirmed that "loaded" (when describing the STYP_* flags)
means that raw data appears in their COFF file for that section.  I
assume such sections would never have a section size of 0.

And "allocated" means space set aside when the program is inserted in
memory for running.

BTW, the "allocated" term agrees with the O'Reilly COFF book definitions.
Specifically:

"Allocation deals with the application's memory space.  If a section
is allocated, that means it uses a part of the memory space available
to the application; no other section can occupy this memory space.
The linker output map shows the memory space occupied by the section."


The O'Reilly book has this to say about "loaded":

"Section type information is also used by the system loader.
Depending on the section type, its raw data are either loaded into
memory or not loaded."


Thus, in most cases a section that is loaded would normally be
allocated as well.



So, back to my original problem:

The STYP_COPY section is not allocated, but is loaded.

TI says that this means data appears in the file but is not written to
memory by the system loader:

"The copy section is present in the output COFF file, but doesn't
occupy space in target memory. It can still reference variables in
other sections, etc."

GNU binutils says the same words, but since "loaded" there means that
it is written to memory, the loader is also to write it to memory.
I'm not sure how the system loader is supposed to handle it.  I guess
it writes to memory without allocating (i.e., without reserving that
hunk of memory for that application).  I imagine that there is limited
application for this in a typical unix userland application.

Note that this is different than an overlay (STYP_OVER) which is
relocated, but not allocated nor loaded to memory.
 

In either case, I now think my original patch to coffcode.h is still
valid.  GNU binutils doesn't handle STYP_COPY nor STYP_DSECT at all,
and it looks like STYP_PAD is handled incorrectly.

 - For STYP_COPY & STYP_DSECT flags the patch clears the SEC_ALLOC
   bit.  These STYP_* flags were previously unhandled.

 - Also it fixes how STYP_PAD is treated... namely, it sets SEC_LOAD
   (instead of setting sec_flags to 0 improperly, IMO) unless
   SEC_NEVER_LOAD was set.

 - It's possible that for TI specific COFF handling, we can add
   code to clear SEC_LOAD for STYP_COPY as well.  But my patch
   for the general coffcode.h does not include that.

For reference, here are the comments from include/coff/internal.h
associated with the affected flags:

#define STYP_DSECT       (0x0001)       /* "dummy":  relocated only*/
#define STYP_PAD         (0x0008)       /* "padding": not allocated, not relocated, loaded */
#define STYP_COPY        (0x0010)       /* "copy": for decision function used by field update;
                                           not allocated, not relocated, loaded;
                                           reloc & lineno entries processed normally */

Again, here is the patch.  Further comments?

from  John Hein  <jhein@timing.com>

	* bfd/coffcode.h [!COFF_WITH_PE] (styp_to_sec_flags): handle
	STYP_COPY & STYP_DSECT flags. Fix STYP_PAD to set SEC_LOAD as
	appropriate. These changes match comments in
	include/coff/internal.h

--- bfd/coffcode.h.orig	Fri Apr  4 17:41:35 2003
+++ bfd/coffcode.h	Fri Apr  4 17:43:41 2003
@@ -644,8 +644,10 @@
       sec_flags |= SEC_DEBUGGING;
 #endif
     }
+  else if ((styp_flags & STYP_PAD) && (sec_flags & SEC_NEVER_LOAD))
+	sec_flags = 0;
   else if (styp_flags & STYP_PAD)
-    sec_flags = 0;
+	sec_flags = SEC_LOAD;
   else if (strcmp (name, _TEXT) == 0)
     {
       if (sec_flags & SEC_NEVER_LOAD)
@@ -702,6 +704,9 @@
   if (styp_flags & STYP_OTHER_LOAD)
     sec_flags = (SEC_LOAD | SEC_ALLOC);
 #endif /* STYP_SDATA */
+
+  if (styp_flags & (STYP_COPY | STYP_DSECT))
+	sec_flags &= ~SEC_ALLOC);
 
 #if defined (COFF_LONG_SECTION_NAMES) && defined (COFF_SUPPORT_GNU_LINKONCE)
   /* As a GNU extension, if the name begins with .gnu.linkonce, we