darwin LTO broken under Xcode 3.2.6/4.0

darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
the darwin assembler which effectively breaks LTO support for darwin. The design
of LTO on darwin was based on the fact that mach-o object files tolerated additional
sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
appears to be strictly counting sections and objecting when these exceed 255. This
breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
change, our only recourse would be to resort to an elf object container for the lto
sections within the mach-o files (introducing an undesired dependency on libelf for
FSF gcc on darwin). My understanding was that the lto design did not allow the number
of sections required in the lto files to be reduced.
              Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Ian Lance Taylor]

Jack Howarth <howarth@bromo.med.uc.edu> writes:

>    With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> the darwin assembler which effectively breaks LTO support for darwin. The design
> of LTO on darwin was based on the fact that mach-o object files tolerated additional
> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> appears to be strictly counting sections and objecting when these exceed 255. This
> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> change, our only recourse would be to resort to an elf object container for the lto
> sections within the mach-o files (introducing an undesired dependency on libelf for
> FSF gcc on darwin). My understanding was that the lto design did not allow the number
> of sections required in the lto files to be reduced.

We no longer use libelf on any system.

This is largely controlled by the simple-object interface.  It should be
straightforward to change the way that simple-object works with Mach-O
without changing the simple-object interface to the rest of gcc.  Then
we could change the gcc output to, e.g., put all the information in a
single Mach-O section.

I suppose we should first ask Apple why they made the change and whether
it was intentional.

Ian

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 12, 2011, at 8:17 PM, Jack Howarth wrote:

>   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> the darwin assembler which effectively breaks LTO support for darwin. The design
> of LTO on darwin was based on the fact that mach-o object files tolerated additional
> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> appears to be strictly counting sections and objecting when these exceed 255. This
> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> change, our only recourse would be to resort to an elf object container for the lto
> sections within the mach-o files (introducing an undesired dependency on libelf for
> FSF gcc on darwin). My understanding was that the lto design did not allow the number
> of sections required in the lto files to be reduced.

Hi Jack,

Please file a bug against the apple bug tracker explaining exactly what you need to work with some example .o files.

-Chris

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sat, Mar 12, 2011 at 09:34:01PM -0800, Chris Lattner wrote:
> 
> On Mar 12, 2011, at 8:17 PM, Jack Howarth wrote:
> 
> >   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> > the darwin assembler which effectively breaks LTO support for darwin. The design
> > of LTO on darwin was based on the fact that mach-o object files tolerated additional
> > sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> > appears to be strictly counting sections and objecting when these exceed 255. This
> > breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> > to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> > change, our only recourse would be to resort to an elf object container for the lto
> > sections within the mach-o files (introducing an undesired dependency on libelf for
> > FSF gcc on darwin). My understanding was that the lto design did not allow the number
> > of sections required in the lto files to be reduced.
> 
> Hi Jack,
> 
> Please file a bug against the apple bug tracker explaining exactly what you need to work with some example .o files.
> 
> -Chris

Chris,
   I have already opened Problem ID: 9126174 on radar and uploaded the builtin-attr-1.s assembly file
from the failed...

/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/xgcc -B/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/ /sw/src/fink.build/gcc46-4.6.0-1000/gcc-4.6-20110311/gcc/testsuite/gcc.dg/torture/builtin-attr-1.c -O2 -flto -flto-partition=none -ffast-math -lm --save-temps -v -m32 -o builtin-attr-1.exe

linkage. The failure occurs as...

builtin-attr-1.s:12528:FATAL:too many sections (maximum 255)

when 'as -arch i386 -force_cpusubtype_ALL -o builtin-attr-1.o builtin-attr-1.s' is executed.
This same assembly file works fine with 'as' from Xcode 3.2.5. Also the 12528th line happens
to be just after the 253rd GNU_LTO section which makes sense as there is a  __TEXT,__text_startup
 and __TEXT,__eh_frame section as well. If I truncate the assembly file at 253 GNU_LTO sections
it is assembled without complaint. Nick believed that the assembler was looking for symbols in
each section and verifying the section count that way. I don't believe that is true and that a simple
count of all sections is being done instead. If true, it prohibits us from using the current approach
of placing all of the GNU_LTO sections at the end of the assembly. Since they don't contain symbols
this has never been a problem in the past.
                Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jan Hubicka]

>    With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> the darwin assembler which effectively breaks LTO support for darwin. The design
> of LTO on darwin was based on the fact that mach-o object files tolerated additional
> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> appears to be strictly counting sections and objecting when these exceed 255. This
> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> change, our only recourse would be to resort to an elf object container for the lto
> sections within the mach-o files (introducing an undesired dependency on libelf for
> FSF gcc on darwin). My understanding was that the lto design did not allow the number
> of sections required in the lto files to be reduced.

If the problem is not fixed, we could always pack all the LTO sections into one section containing
our own subsections.

Honza
>               Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 12:39:26PM +0100, Jan Hubicka wrote:
> >    With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> > the darwin assembler which effectively breaks LTO support for darwin. The design
> > of LTO on darwin was based on the fact that mach-o object files tolerated additional
> > sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> > appears to be strictly counting sections and objecting when these exceed 255. This
> > breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> > to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> > change, our only recourse would be to resort to an elf object container for the lto
> > sections within the mach-o files (introducing an undesired dependency on libelf for
> > FSF gcc on darwin). My understanding was that the lto design did not allow the number
> > of sections required in the lto files to be reduced.
> 
> If the problem is not fixed, we could always pack all the LTO sections into one section containing
> our own subsections.
> 
> Honza

Jan,
  If this could be done without resorting to other container types (like elf), it might be
the wisest approach for the long run. I've read through the mach-o documentation and it
seems rather vague on the section limits. Even if Apple fixes Xcode (which likley won't
happen for 6-9 months at best), we always we have to worry that they will break this 
'feature' somewhere else in their tool chain. Better to follow the strictest possible reading
of mach-o object format to protect ourselves from overzealous Apple interns.
> >               Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 8:38 AM, Jack Howarth wrote:

> On Sun, Mar 13, 2011 at 12:39:26PM +0100, Jan Hubicka wrote:
>>>   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
>>> the darwin assembler which effectively breaks LTO support for darwin. The design
>>> of LTO on darwin was based on the fact that mach-o object files tolerated additional
>>> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
>>> appears to be strictly counting sections and objecting when these exceed 255. This
>>> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
>>> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
>>> change, our only recourse would be to resort to an elf object container for the lto
>>> sections within the mach-o files (introducing an undesired dependency on libelf for
>>> FSF gcc on darwin). My understanding was that the lto design did not allow the number
>>> of sections required in the lto files to be reduced.
>> 
>> If the problem is not fixed, we could always pack all the LTO sections into one section containing
>> our own subsections.
>> 
>> Honza
> 
> Jan,
>  If this could be done without resorting to other container types (like elf), it might be
> the wisest approach for the long run. I've read through the mach-o documentation and it
> seems rather vague on the section limits. Even if Apple fixes Xcode (which likley won't
> happen for 6-9 months at best), we always we have to worry that they will break this 
> 'feature' somewhere else in their tool chain. Better to follow the strictest possible reading
> of mach-o object format to protect ourselves from overzealous Apple interns.

Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.

-Chris

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 11:19:13AM -0700, Chris Lattner wrote:
> 
> On Mar 13, 2011, at 8:38 AM, Jack Howarth wrote:
> 
> > On Sun, Mar 13, 2011 at 12:39:26PM +0100, Jan Hubicka wrote:
> >>>   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> >>> the darwin assembler which effectively breaks LTO support for darwin. The design
> >>> of LTO on darwin was based on the fact that mach-o object files tolerated additional
> >>> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> >>> appears to be strictly counting sections and objecting when these exceed 255. This
> >>> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> >>> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> >>> change, our only recourse would be to resort to an elf object container for the lto
> >>> sections within the mach-o files (introducing an undesired dependency on libelf for
> >>> FSF gcc on darwin). My understanding was that the lto design did not allow the number
> >>> of sections required in the lto files to be reduced.
> >> 
> >> If the problem is not fixed, we could always pack all the LTO sections into one section containing
> >> our own subsections.
> >> 
> >> Honza
> > 
> > Jan,
> >  If this could be done without resorting to other container types (like elf), it might be
> > the wisest approach for the long run. I've read through the mach-o documentation and it
> > seems rather vague on the section limits. Even if Apple fixes Xcode (which likley won't
> > happen for 6-9 months at best), we always we have to worry that they will break this 
> > 'feature' somewhere else in their tool chain. Better to follow the strictest possible reading
> > of mach-o object format to protect ourselves from overzealous Apple interns.
> 
> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
> 
> -Chris

Chris,
   Is there any documentation or example code on how to properly use subsections in mach-o?
My fear is that we are moving from one poorly documented technique to another which may well
have it own slate of hidden bugs.
          Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:

>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
>> 
>> -Chris
> 
> Chris,
>   Is there any documentation or example code on how to properly use subsections in mach-o?
> My fear is that we are moving from one poorly documented technique to another which may well
> have it own slate of hidden bugs.

I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.

-Chris

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 11:55 AM, Chris Lattner wrote:
> On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:
> 
>>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
>>> 
>>> -Chris
>> 
>> Chris,
>>  Is there any documentation or example code on how to properly use subsections in mach-o?
>> My fear is that we are moving from one poorly documented technique to another which may well
>> have it own slate of hidden bugs.
> 
> I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.

Sorry, I actually mean 255 of course, because of the NO_SECT sentinel.  Here are the relevant bits from nlist.h.  I'm not sure how you expect the toolchain to store more than 256 sections in a uint8_t.

This is not the conspiracy you are looking for.

-Chris



/*
 * Format of a symbol table entry of a Mach-O file for 32-bit architectures.
 * Modified from the BSD format.  The modifications from the original format
 * were changing n_other (an unused field) to n_sect and the addition of the
 * N_SECT type.  These modifications are required to support symbols in a larger
 * number of sections not just the three sections (text, data and bss) in a BSD
 * file.
 */
struct nlist {
        union {
#ifndef __LP64__
                char *n_name;   /* for use when in-core */
#endif
                int32_t n_strx; /* index into the string table */
        } n_un;
        uint8_t n_type;         /* type flag, see below */
        uint8_t n_sect;         /* section number or NO_SECT */
        int16_t n_desc;         /* see <mach-o/stab.h> */
        uint32_t n_value;       /* value of this symbol (or stab offset) */
};

/*
 * This is the symbol table entry structure for 64-bit architectures.
 */
struct nlist_64 {
    union {
        uint32_t  n_strx; /* index into the string table */
    } n_un;
    uint8_t n_type;        /* type flag, see below */
    uint8_t n_sect;        /* section number or NO_SECT */
    uint16_t n_desc;       /* see <mach-o/stab.h> */
    uint64_t n_value;      /* value of this symbol (or stab offset) */
};

/*
 * If the type is N_SECT then the n_sect field contains an ordinal of the
 * section the symbol is defined in.  The sections are numbered from 1 and 
 * refer to sections in order they appear in the load commands for the file
 * they are in.  This means the same ordinal may very well refer to different
 * sections in different files.
 *
 * The n_value field for all symbol table entries (including N_STAB's) gets
 * updated by the link editor based on the value of it's n_sect field and where
 * the section n_sect references gets relocated.  If the value of the n_sect 
 * field is NO_SECT then it's n_value field is not changed by the link editor.
 */
#define NO_SECT         0       /* symbol is not in any section */
#define MAX_SECT        255     /* 1 thru 255 inclusive */

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 11:59:33AM -0700, Chris Lattner wrote:
> On Mar 13, 2011, at 11:55 AM, Chris Lattner wrote:
> > On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:
> > 
> >>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
> >>> 
> >>> -Chris
> >> 
> >> Chris,
> >>  Is there any documentation or example code on how to properly use subsections in mach-o?
> >> My fear is that we are moving from one poorly documented technique to another which may well
> >> have it own slate of hidden bugs.
> > 
> > I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.
> 
> Sorry, I actually mean 255 of course, because of the NO_SECT sentinel.  Here are the relevant bits from nlist.h.  I'm not sure how you expect the toolchain to store more than 256 sections in a uint8_t.
> 
> This is not the conspiracy you are looking for.
> 
> -Chris

Chris,
   Nick's most recent comments to me on this were...

mach-o has always had a restriction on the number of sections.  The mach-o nlist struct has a one byte n_sect field which is the index of the section that symbol is in.  Previously,
+the assembler did not validate that the n_sect field was not overflowed.  Now it does validate.

He left us with the impression that only sections actually containing symbols were validated
and that our original design (based on his recommendations) of placing all of the symbol-less
GNU_LTO sections at the end of the file was still okay.
             Jack

> 
> 
> 
> /*
>  * Format of a symbol table entry of a Mach-O file for 32-bit architectures.
>  * Modified from the BSD format.  The modifications from the original format
>  * were changing n_other (an unused field) to n_sect and the addition of the
>  * N_SECT type.  These modifications are required to support symbols in a larger
>  * number of sections not just the three sections (text, data and bss) in a BSD
>  * file.
>  */
> struct nlist {
>         union {
> #ifndef __LP64__
>                 char *n_name;   /* for use when in-core */
> #endif
>                 int32_t n_strx; /* index into the string table */
>         } n_un;
>         uint8_t n_type;         /* type flag, see below */
>         uint8_t n_sect;         /* section number or NO_SECT */
>         int16_t n_desc;         /* see <mach-o/stab.h> */
>         uint32_t n_value;       /* value of this symbol (or stab offset) */
> };
> 
> /*
>  * This is the symbol table entry structure for 64-bit architectures.
>  */
> struct nlist_64 {
>     union {
>         uint32_t  n_strx; /* index into the string table */
>     } n_un;
>     uint8_t n_type;        /* type flag, see below */
>     uint8_t n_sect;        /* section number or NO_SECT */
>     uint16_t n_desc;       /* see <mach-o/stab.h> */
>     uint64_t n_value;      /* value of this symbol (or stab offset) */
> };
> 
> /*
>  * If the type is N_SECT then the n_sect field contains an ordinal of the
>  * section the symbol is defined in.  The sections are numbered from 1 and 
>  * refer to sections in order they appear in the load commands for the file
>  * they are in.  This means the same ordinal may very well refer to different
>  * sections in different files.
>  *
>  * The n_value field for all symbol table entries (including N_STAB's) gets
>  * updated by the link editor based on the value of it's n_sect field and where
>  * the section n_sect references gets relocated.  If the value of the n_sect 
>  * field is NO_SECT then it's n_value field is not changed by the link editor.
>  */
> #define NO_SECT         0       /* symbol is not in any section */
> #define MAX_SECT        255     /* 1 thru 255 inclusive */

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 11:55:02AM -0700, Chris Lattner wrote:
> 
> On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:
> 
> >> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
> >> 
> >> -Chris
> > 
> > Chris,
> >   Is there any documentation or example code on how to properly use subsections in mach-o?
> > My fear is that we are moving from one poorly documented technique to another which may well
> > have it own slate of hidden bugs.
> 
> I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.
> 
> -Chris

Chris,
  The mach-o reference has a cryptic mention of MH_SUBSECTIONS_VIA_SYMBOLS in

http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html

MH_SUBSECTIONS_VIA_SYMBOLS—The sections of the object file can be divided into individual blocks. These blocks are dead-stripped if they are not used by other code. See Linking for details.

Although I don't think this helps us at all because none of the GNU_LTO sections contain symbols. This is why I
am still perplexed by this assembler change. We followed the rules Nick suggested of placing all of these GNU_LTO
sections at the end of the file and none contain symbols. In that case, it would really seem that the assembler is
now blindly counting sections without respect to whether any of them actually contain symbols.
          Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 12:05 PM, Jack Howarth wrote:

> On Sun, Mar 13, 2011 at 11:55:02AM -0700, Chris Lattner wrote:
>> 
>> On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:
>> 
>>>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
>>>> 
>>>> -Chris
>>> 
>>> Chris,
>>>  Is there any documentation or example code on how to properly use subsections in mach-o?
>>> My fear is that we are moving from one poorly documented technique to another which may well
>>> have it own slate of hidden bugs.
>> 
>> I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.
>> 
>> -Chris
> 
> Chris,
>  The mach-o reference has a cryptic mention of MH_SUBSECTIONS_VIA_SYMBOLS in
> 
> http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html
> 
> MH_SUBSECTIONS_VIA_SYMBOLS—The sections of the object file can be divided into individual blocks. These blocks are dead-stripped if they are not used by other code. See Linking for details.

Right, "subsections via symbols" is the concept of slicing up a section into "atoms" based on public labels.  The linker uses this to do dead code elimination etc.  I don't think this helps you at all though.

I don't understand why you can't put arbitrary data into a single section.  This is what the -sectcreate flag to the linker does, so it is simple, already supported, and tested.

-Chris

> 
> Although I don't think this helps us at all because none of the GNU_LTO sections contain symbols. This is why I
> am still perplexed by this assembler change. We followed the rules Nick suggested of placing all of these GNU_LTO
> sections at the end of the file and none contain symbols. In that case, it would really seem that the assembler is
> now blindly counting sections without respect to whether any of them actually contain symbols.
>          Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 12:47:22PM -0700, Chris Lattner wrote:
> 
> On Mar 13, 2011, at 12:05 PM, Jack Howarth wrote:
> 
> > On Sun, Mar 13, 2011 at 11:55:02AM -0700, Chris Lattner wrote:
> >> 
> >> On Mar 13, 2011, at 11:26 AM, Jack Howarth wrote:
> >> 
> >>>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
> >>>> 
> >>>> -Chris
> >>> 
> >>> Chris,
> >>>  Is there any documentation or example code on how to properly use subsections in mach-o?
> >>> My fear is that we are moving from one poorly documented technique to another which may well
> >>> have it own slate of hidden bugs.
> >> 
> >> I'm not sure what you mean here Jack.  ld64 is open source on the darwin page, and if you have all the developer bits installed, the format is "documented" in /usr/include/mach-o/.  It's pretty clear all around that you can only have 256 sections.
> >> 
> >> -Chris
> > 
> > Chris,
> >  The mach-o reference has a cryptic mention of MH_SUBSECTIONS_VIA_SYMBOLS in
> > 
> > http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html
> > 
> > MH_SUBSECTIONS_VIA_SYMBOLS—The sections of the object file can be divided into individual blocks. These blocks are dead-stripped if they are not used by other code. See Linking for details.
> 
> Right, "subsections via symbols" is the concept of slicing up a section into "atoms" based on public labels.  The linker uses this to do dead code elimination etc.  I don't think this helps you at all though.
> 
> I don't understand why you can't put arbitrary data into a single section.  This is what the -sectcreate flag to the linker does, so it is simple, already supported, and tested.

Chris,
   Sure, aasuming we can find a victim to do the unncessary coding. Again, Nick left us with the impression that n_sect was
only sanity checked in the case of symbols being present in the sections. This isn't the case. I've used the FSF gcc
LTO for very large projects as well as routinely bootstraping FSF gcc with LTO an it has worked perfectly under Xcode
3.2.5. So this isn't a fragile situation as we never output more than 255 sections with symbols in the final linkage.
The annoyance witnessed here is that Apple isn't following it's own mach-o documentation as it exists. However I agree
it is in the best interest for us to recode this in order to protect ourselves from further improper Xcode changes
in the future. It will be interesting to see if this ends up causing misery for other developers who have relied 
on Apple's mach-o documentation being honored for their own coding of additional non-symbol containing sections.
              Jack
> 
> -Chris
> 
> > 
> > Although I don't think this helps us at all because none of the GNU_LTO sections contain symbols. This is why I
> > am still perplexed by this assembler change. We followed the rules Nick suggested of placing all of these GNU_LTO
> > sections at the end of the file and none contain symbols. In that case, it would really seem that the assembler is
> > now blindly counting sections without respect to whether any of them actually contain symbols.
> >          Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

Chris,
   Could you clarify the following question from PR48086?

(In reply to comment #11)
> (In reply to comment #10)
> > The easiest way to fix this is maybe to just have more than one GNU_LTO
> > segment. AFAIU the limit of 255 sections is a limit per segment. It is not
> > difficult to have multiple GNU_LTO segments, I could even hack that in before
> > GCC 4.6.

hmmm. maybe not .. re-reading:

n_sect 
An integer specifying the number of the section that this symbol can be found
in, or NO_SECT if the 
symbol is not to be found in any section of this image.  The sections are
contiguously numbered across 
segments, starting from 1, according to the order they appear in the LC_SEGMENT
load commands. 

contiguously numbered across segments; seems to indicate that the section count
is per file - not per segment.

             Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jan Hubicka]

> 
> On Mar 13, 2011, at 8:38 AM, Jack Howarth wrote:
> 
> > On Sun, Mar 13, 2011 at 12:39:26PM +0100, Jan Hubicka wrote:
> >>>   With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> >>> the darwin assembler which effectively breaks LTO support for darwin. The design
> >>> of LTO on darwin was based on the fact that mach-o object files tolerated additional
> >>> sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> >>> appears to be strictly counting sections and objecting when these exceed 255. This
> >>> breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> >>> to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> >>> change, our only recourse would be to resort to an elf object container for the lto
> >>> sections within the mach-o files (introducing an undesired dependency on libelf for
> >>> FSF gcc on darwin). My understanding was that the lto design did not allow the number
> >>> of sections required in the lto files to be reduced.
> >> 
> >> If the problem is not fixed, we could always pack all the LTO sections into one section containing
> >> our own subsections.
> >> 
> >> Honza
> > 
> > Jan,
> >  If this could be done without resorting to other container types (like elf), it might be
> > the wisest approach for the long run. I've read through the mach-o documentation and it
> > seems rather vague on the section limits. Even if Apple fixes Xcode (which likley won't
> > happen for 6-9 months at best), we always we have to worry that they will break this 
> > 'feature' somewhere else in their tool chain. Better to follow the strictest possible reading
> > of mach-o object format to protect ourselves from overzealous Apple interns.
> 
> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.

Interesting, so there is no -ffunction-section type tricks at darwin?

Pickling everything into single section seems indeed like easiest approach here.
The question is who will volunteer to implement it ;)

Honza
> 
> -Chris

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 12:07 PM, Jan Hubicka wrote:

>> 
>> Yes, I agree that this is a better solution.  This error was put into the linker to detect some overflow conditions for part of the code that expected the section number to only be a byte.  It is likely that "things worked" only out of luck before.
> 
> Interesting, so there is no -ffunction-section type tricks at darwin?

Correct, darwin doesn't support that flag in a useful way.  Instead, macho has a (poorly defined and understood) concept of "atoms" that the linker uses to slice things up.

-Chris

> 
> Pickling everything into single section seems indeed like easiest approach here.
> The question is who will volunteer to implement it ;)
> 
> Honza

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 12:39:26PM +0100, Jan Hubicka wrote:
> >    With release of Xcode 3.2.6/4.0 this week, an unfortunate change was made to
> > the darwin assembler which effectively breaks LTO support for darwin. The design
> > of LTO on darwin was based on the fact that mach-o object files tolerated additional
> > sections as long as they didin't contain symbols. With Xcode 3.2.6/4.0, the assembler
> > appears to be strictly counting sections and objecting when these exceed 255. This
> > breaks huge sections of the lto testsuite and prevents larger projects like xplor-nih
> > to compile if Xcode 3.2.6/4.0 is installed. I am afraid that unless Apple reverts this
> > change, our only recourse would be to resort to an elf object container for the lto
> > sections within the mach-o files (introducing an undesired dependency on libelf for
> > FSF gcc on darwin). My understanding was that the lto design did not allow the number
> > of sections required in the lto files to be reduced.
> 
> If the problem is not fixed, we could always pack all the LTO sections into one section containing
> our own subsections.
> 
> Honza

Jan,
   It appears this change is highly unlikely to be fixed. Chris Lattner said...

> Jack, various datastructures in the toolchain and the .o file only store the section number as a byte.  
> Allowing it to overflow just masks bugs because section id's will be silently truncated.  You 
> should fix the GCC LTO model to store one big section with its own subcontents if you want more granularity.

So lto-object.c needs a rewrite to use only a single section for GNU_LTO with subsections.
Unfortunately I can't find any documentation for using subsections in mach-o which may imply we will
be forced to use an elf container to obtain those, no?
                Jack

> >               Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Ian Lance Taylor]

Jack Howarth <howarth@bromo.med.uc.edu> writes:

> So lto-object.c needs a rewrite to use only a single section for GNU_LTO with subsections.
> Unfortunately I can't find any documentation for using subsections in mach-o which may imply we will
> be forced to use an elf container to obtain those, no?

We can format the information in the section any way we like.  No reason
to fix on subsections or on ELF, though both are certainly options.  The
important characteristics are: 1) the Mach-O assembler has to accept it;
2) the simple-object interface in libiberty has to read it.

Ian

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 11:53:59AM -0700, Ian Lance Taylor wrote:
> Jack Howarth <howarth@bromo.med.uc.edu> writes:
> 
> > So lto-object.c needs a rewrite to use only a single section for GNU_LTO with subsections.
> > Unfortunately I can't find any documentation for using subsections in mach-o which may imply we will
> > be forced to use an elf container to obtain those, no?
> 
> We can format the information in the section any way we like.  No reason
> to fix on subsections or on ELF, though both are certainly options.  The
> important characteristics are: 1) the Mach-O assembler has to accept it;
> 2) the simple-object interface in libiberty has to read it.
> 
> Ian

Ian,
    Are there any places in the current gcc sources which would give us examples
of using the simple-object interface of libiberty in a manner similar to what we
need for storing the lto data in the mach-o section? Or perhaps in an older FSF 
gcc release?
         Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Ian Lance Taylor]

Jack Howarth <howarth@bromo.med.uc.edu> writes:

> On Sun, Mar 13, 2011 at 11:53:59AM -0700, Ian Lance Taylor wrote:
>> Jack Howarth <howarth@bromo.med.uc.edu> writes:
>> 
>> > So lto-object.c needs a rewrite to use only a single section for GNU_LTO with subsections.
>> > Unfortunately I can't find any documentation for using subsections in mach-o which may imply we will
>> > be forced to use an elf container to obtain those, no?
>> 
>> We can format the information in the section any way we like.  No reason
>> to fix on subsections or on ELF, though both are certainly options.  The
>> important characteristics are: 1) the Mach-O assembler has to accept it;
>> 2) the simple-object interface in libiberty has to read it.
>> 
>> Ian
>
> Ian,
>     Are there any places in the current gcc sources which would give us examples
> of using the simple-object interface of libiberty in a manner similar to what we
> need for storing the lto data in the mach-o section? Or perhaps in an older FSF 
> gcc release?

I think you are overthinking this.  What LTO needs to do is really
simple: store byte sequences with names.  Anything which lets you do
that will work.  You need to write it out in a way that the assembler
will accept; simple-object is not involved at that point.  And you need
to modify simple-object to read it.  And then you need to modify
simple-object to write it, for the plugin to use.

Ian

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Mon, Mar 14, 2011 at 02:08:47PM -0700, Ian Lance Taylor wrote:
> 
> I think you are overthinking this.  What LTO needs to do is really
> simple: store byte sequences with names.  Anything which lets you do
> that will work.  You need to write it out in a way that the assembler
> will accept; simple-object is not involved at that point.  And you need
> to modify simple-object to read it.  And then you need to modify
> simple-object to write it, for the plugin to use.
> 
> Ian

Ian,
   So do we want to create a new simple-object-lto.c to handle this, so that the
code can be used from other object formats, or should we just shoehorn the new code
into simple-object-mach-o.c? I am unclear if any target besides darwin is currently
using lto/lto-object.c The comment in that file...

   FIXME: This needs to be kept in sync with darwin.c.  */

made me wonder if that was the case now that cygwin/mingw have plugin support.
                    Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Steven Bosscher]

(sorry Chris, I forgot the list)

On Mar 13, 2011, at 11:59 AM, Chris Lattner wrote:

> Sorry, I actually mean 255 of course, because of the NO_SECT
> sentinel.  Here are the relevant bits from nlist.h.  I'm not
> sure how you expect the toolchain to store more than 256
> sections in a uint8_t.

How self-righteous, and misinformed.

No-one is expecting to store >256 in a uint8_t. The structures you
quote only apply to symbol tables, which are references into sections
in Mach-O. But not all sections have symbols. None of the sections in
the GNU_LTO segment have symbols.

The documentation you should be quoting is the part about Mach-O
loader commands, and there is no limit AFAICT on the number of
LC_SEGMENT/LC_SEGMENT_64 loader commands. There is also nothing in the
segment command structures that suggests a limit of 255 sections.

Besides, it worked before, which suggests that, well, it worked.

This is not the limitation of Mach-O you are looking for.

Ciao!
Steven

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Chris Lattner]

On Mar 13, 2011, at 12:42 PM, Steven Bosscher wrote:

> (sorry Chris, I forgot the list)
> 
> On Mar 13, 2011, at 11:59 AM, Chris Lattner wrote:
> 
>> Sorry, I actually mean 255 of course, because of the NO_SECT
>> sentinel.  Here are the relevant bits from nlist.h.  I'm not
>> sure how you expect the toolchain to store more than 256
>> sections in a uint8_t.
> 
> How self-righteous, and misinformed.
> 
> No-one is expecting to store >256 in a uint8_t. The structures you
> quote only apply to symbol tables, which are references into sections
> in Mach-O. But not all sections have symbols. None of the sections in
> the GNU_LTO segment have symbols.

That data structure only has to do with symbol tables.  However, various tools that work on MachO have the exact same limitation in their internal data structures, since "there can't be more than 256 sections".

I'm sorry that the toolchain doesn't work the way you'd like it to, but the right answer is still to plop whatever metadata you guys want into a single section.

-Chris


> 
> The documentation you should be quoting is the part about Mach-O
> loader commands, and there is no limit AFAICT on the number of
> LC_SEGMENT/LC_SEGMENT_64 loader commands. There is also nothing in the
> segment command structures that suggests a limit of 255 sections.
> 
> Besides, it worked before, which suggests that, well, it worked.
> 
> This is not the limitation of Mach-O you are looking for.
> 
> Ciao!
> Steven

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 08:42:58PM +0100, Steven Bosscher wrote:
> (sorry Chris, I forgot the list)
> 
> On Mar 13, 2011,@11:59 AM, Chris Lattner wrote:
> 
> > Sorry, I actually mean 255 of course, because of the NO_SECT
> > sentinel.  Here are the relevant bits from nlist.h.  I'm not
> > sure how you expect the toolchain to store more than 256
> > sections in a uint8_t.
> 
> How self-righteous, and misinformed.
> 
> No-one is expecting to store >256 in a uint8_t. The structures you
> quote only apply to symbol tables, which are references into sections
> in Mach-O. But not all sections have symbols. None of the sections in
> the GNU_LTO segment have symbols.
> 
> The documentation you should be quoting is the part about Mach-O
> loader commands, and there is no limit AFAICT on the number of
> LC_SEGMENT/LC_SEGMENT_64 loader commands. There is also nothing in the
> segment command structures that suggests a limit of 255 sections.
> 
> Besides, it worked before, which suggests that, well, it worked.
> 
> This is not the limitation of Mach-O you are looking for.
> 
> Ciao!
> Steven

Steven,
   While I agree that we are likely correct in the details of this
problem, I suspect we will be unsuccessful in getting Apple to revert
this change. Even if they do so, it is likely to be for Xcode 4.0.1
only (since the 3.2.x series is being depreciated) and at best we
might see the fix in 6-9 months if not later. It is far better, despite
the pain, to recode this in order to protect ourselves in the future
from arbitary changes in Xcode. IMHO, one major problem is that Apple has
left mach-o poorly documented in some areas which leaves them the
lattitude to redefine the corner cases like this as they please.
                    Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Steven Bosscher]

On Sun, Mar 13, 2011 at 9:03 PM, Jack Howarth <howarth@bromo.med.uc.edu> wrote:
> On Sun, Mar 13, 2011 at 08:42:58PM +0100, Steven Bosscher wrote:
>> (sorry Chris, I forgot the list)
>>
>> On Mar 13, 2011,@11:59 AM, Chris Lattner wrote:
>>
>> > Sorry, I actually mean 255 of course, because of the NO_SECT
>> > sentinel.  Here are the relevant bits from nlist.h.  I'm not
>> > sure how you expect the toolchain to store more than 256
>> > sections in a uint8_t.
>>
>> How self-righteous, and misinformed.
>>
>> No-one is expecting to store >256 in a uint8_t. The structures you
>> quote only apply to symbol tables, which are references into sections
>> in Mach-O. But not all sections have symbols. None of the sections in
>> the GNU_LTO segment have symbols.
>>
>> The documentation you should be quoting is the part about Mach-O
>> loader commands, and there is no limit AFAICT on the number of
>> LC_SEGMENT/LC_SEGMENT_64 loader commands. There is also nothing in the
>> segment command structures that suggests a limit of 255 sections.
>>
>> Besides, it worked before, which suggests that, well, it worked.
>>
>> This is not the limitation of Mach-O you are looking for.
>>
>> Ciao!
>> Steven
>
> Steven,
>   While I agree that we are likely correct in the details of this
> problem, I suspect we will be unsuccessful in getting Apple to revert
> this change. Even if they do so, it is likely to be for Xcode 4.0.1
> only (since the 3.2.x series is being depreciated) and at best we
> might see the fix in 6-9 months if not later. It is far better, despite
> the pain, to recode this in order to protect ourselves in the future
> from arbitary changes in Xcode. IMHO, one major problem is that Apple has
> left mach-o poorly documented in some areas which leaves them the
> lattitude to redefine the corner cases like this as they please.
>                    Jack
>

Hi,

I agree it is probably better to re-code things, but that will be
impossible do before GCC 4.6 goes out.

We have to make a decision: keep the LTO support for Mach-O in for now
and recommend a non-recent Xcode, disable it for the GCC 4.6 release
and for the GCC 4.5.x series.

Ciao!
Steven

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 09:38:06PM +0100, Steven Bosscher wrote:
> On Sun, Mar 13, 2011 at 9:03 PM, Jack Howarth <howarth@bromo.med.uc.edu> wrote:
> > On Sun, Mar 13, 2011 at 08:42:58PM +0100, Steven Bosscher wrote:
> >> (sorry Chris, I forgot the list)
> >>
> >> On Mar 13, 2011,@11:59 AM, Chris Lattner wrote:
> >>
> >> > Sorry, I actually mean 255 of course, because of the NO_SECT
> >> > sentinel.  Here are the relevant bits from nlist.h.  I'm not
> >> > sure how you expect the toolchain to store more than 256
> >> > sections in a uint8_t.
> >>
> >> How self-righteous, and misinformed.
> >>
> >> No-one is expecting to store >256 in a uint8_t. The structures you
> >> quote only apply to symbol tables, which are references into sections
> >> in Mach-O. But not all sections have symbols. None of the sections in
> >> the GNU_LTO segment have symbols.
> >>
> >> The documentation you should be quoting is the part about Mach-O
> >> loader commands, and there is no limit AFAICT on the number of
> >> LC_SEGMENT/LC_SEGMENT_64 loader commands. There is also nothing in the
> >> segment command structures that suggests a limit of 255 sections.
> >>
> >> Besides, it worked before, which suggests that, well, it worked.
> >>
> >> This is not the limitation of Mach-O you are looking for.
> >>
> >> Ciao!
> >> Steven
> >
> > Steven,
> >   While I agree that we are likely correct in the details of this
> > problem, I suspect we will be unsuccessful in getting Apple to revert
> > this change. Even if they do so, it is likely to be for Xcode 4.0.1
> > only (since the 3.2.x series is being depreciated) and at best we
> > might see the fix in 6-9 months if not later. It is far better, despite
> > the pain, to recode this in order to protect ourselves in the future
> > from arbitary changes in Xcode. IMHO, one major problem is that Apple has
> > left mach-o poorly documented in some areas which leaves them the
> > lattitude to redefine the corner cases like this as they please.
> >                    Jack
> >
> 
> Hi,
> 
> I agree it is probably better to re-code things, but that will be
> impossible do before GCC 4.6 goes out.
> 
> We have to make a decision: keep the LTO support for Mach-O in for now
> and recommend a non-recent Xcode, disable it for the GCC 4.6 release
> and for the GCC 4.5.x series.

Steven,
   Did I misread you message at...

http://gcc.gnu.org/ml/gcc-bugs/2011-03/msg01336.html

as it seems to suggest otherwise? We do need a clear answer from Apple 
about how n_sect is validated though (per segment or per file).
                Jack
> 
> Ciao!
> Steven

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Steven Bosscher]

On Sun, Mar 13, 2011 at 9:41 PM, Jack Howarth <howarth@bromo.med.uc.edu> wrote:
> On Sun, Mar 13, 2011 at 09:38:06PM +0100, Steven Bosscher wrote:
>> I agree it is probably better to re-code things, but that will be
>> impossible do before GCC 4.6 goes out.
>>
>> We have to make a decision: keep the LTO support for Mach-O in for now
>> and recommend a non-recent Xcode, disable it for the GCC 4.6 release
>> and for the GCC 4.5.x series.
>
> Steven,
>   Did I misread you message at...
>
> http://gcc.gnu.org/ml/gcc-bugs/2011-03/msg01336.html
>
> as it seems to suggest otherwise? We do need a clear answer from Apple
> about how n_sect is validated though (per segment or per file).
>                Jack

No, I'm sure Ian is right. For an intermediate object file (.o file)
all sections are stuffed into a single segment, and it also would
appear that the section numbers are per file, not per segment.

Ciao!
Steven

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 09:43:07PM +0100, Steven Bosscher wrote:
> On Sun, Mar 13, 2011 at 9:41 PM, Jack Howarth <howarth@bromo.med.uc.edu> wrote:
> > On Sun, Mar 13, 2011 at 09:38:06PM +0100, Steven Bosscher wrote:
> >> I agree it is probably better to re-code things, but that will be
> >> impossible do before GCC 4.6 goes out.
> >>
> >> We have to make a decision: keep the LTO support for Mach-O in for now
> >> and recommend a non-recent Xcode, disable it for the GCC 4.6 release
> >> and for the GCC 4.5.x series.
> >
> > Steven,
> >   Did I misread you message at...
> >
> > http://gcc.gnu.org/ml/gcc-bugs/2011-03/msg01336.html
> >
> > as it seems to suggest otherwise? We do need a clear answer from Apple
> > about how n_sect is validated though (per segment or per file).
> >                Jack
> 
> No, I'm sure Ian is right. For an intermediate object file (.o file)
> all sections are stuffed into a single segment, and it also would
> appear that the section numbers are per file, not per segment.
> 
> Ciao!
> Steven

Steven,
   In any case, your account is still available on my machine and your welcome
to poke at the problem at your own pace. Remember that Jan Hubicka also has an
account on my machine so he can work the issue as well. If you came up with
a partial mockup and posted it to the bugzilla, Jan could comment and might
even take over the final implementation.
            Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

   Heh, so I guess it's my fault...

> I dug through radar and found the bug that triggered the change to as(1):
>
>  <rdar://problem/7920267> possible assembler bug exposed by LTO
>
> The bug was written (ironically) by Jack Howarth!  At the time 4/28/10), gcc's LTO was putting some LTO section first in the file and that pushed some real sections like __text past 
>the 255.  Then the n_sect field for symbols in the __text section could not contain the section index, resulting in the linker complaining about the .o file.
>
>
>The fix in 7920267 was to just error when the assembler sees the 255th section, regardless of which sections have symbols.  That is what you are seeing in Xocde4.
>
>
>The suggestion of using different segment names will not work.  Segments aren't really used in mach-o object files.   All mach-o object files have exactly one LC_SEGMENT load
>command.  It is final linked images that support multiple  LC_SEGMENT load commands.
>
>It is possible to fix this in Xcode 4.x, but I think a solution that just uses one section for all gcc LTO data (and have your own sub-section scheme) will work with all darwin
> assembler versions.
>
> -Nick

Oh, well. The danger of filing radar bug reports with Apple. On the otherhand, if they weren't so secretive about
posting Xcode previews to the general ADC accounts these days, we would have caught this problem well before
Xcode 4.0 was released. So there is a lot of blame to go around.
         Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

   For clarity, the radar that I filed over which Apple has inflicted this
pain on us was...

------------------------------------------------------------------------------
	 Problem ID: 7920267    possible linker bug exposed by LTO

28-Apr-2010 07:18 PM Jack Howarth:
The FSF gcc developers are implementing their Link Time Optimization support for darwin and it appears to have exposed a possible assembler or linker bug when compiling the aermod benchmark. The failure can be reproduced with the attached test case which executes...

as -arch x86_64 -force_cpusubtype_ALL -o aermod.o aermod.s                
/usr/bin/ld -dynamic -arch x86_64 -macosx_version_min 10.6.3 -weak_reference_mismatches non-weak -o aermod -lcrt1.10.5.o -L./ aermod.o -lgfortran -lgcc_s.10.5 -lgcc_ext.10.5 -no_compact_unwind -lSystem -v

I have provided a bad_link shell script which automatically executes these commands that produce the linker error...

@(#)PROGRAM:ld  PROJECT:ld64-97.2
Library search paths:
	./
	/usr/lib
	/usr/local/lib
Framework search paths:
	/Library/Frameworks/
	/System/Library/Frameworks/
ld: in aermod.o, in section __TEXT,__text reloc 17: local relocation for address 0x000FB35C in section __text does not target section __00000B26


29-Apr-2010 09:35 AM Jack Howarth:
When the assembly files are modified by hand, it has been found that
if the normal output of -flto in FSF gcc is changed from...

LTO sections (the __GNU_LTO stuff)
.text / .data / etc. (all non-LTO sections)
LTO __section_names section

...to a re-ordered .s file of...

.text / .data / etc. (all non-LTO sections)
LTO sections (the __GNU_LTO stuff)
LTO __section_names section

...the linker error disappears. Is this the approach that we need
to take to avoid this issue?

----------------------------------------------------------------------

Apple never responded to this radar with either an analysis or a notification
of a proposed bug fix.
                 Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

   Would someone please correct http://gcc.gnu.org/gcc-4.6/changes.html by deleting
the reference to darwin LTO support. Specifically we should just kill the line...

LTO-support.
Darwin has benefited from ongoing work on LTO; support for this is now stable and enabled by default.

It doesn't merit even mentioning that LTO works under Xcode 3.2.5 because end-users will be
annoyed at their inability to obtain that version if it isn't currently installed.
              Jack

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Gerald Pfeifer]

On Mon, 14 Mar 2011, Jack Howarth wrote:
>    Would someone please correct http://gcc.gnu.org/gcc-4.6/changes.html by deleting
> the reference to darwin LTO support. Specifically we should just kill the line...

Done thusly.

Gerald

Index: changes.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/gcc-4.6/changes.html,v
retrieving revision 1.124
diff -u -r1.124 changes.html
--- changes.html	22 Mar 2011 20:11:00 -0000	1.124
+++ changes.html	2 Apr 2011 22:26:45 -0000
@@ -974,8 +974,6 @@
 	context of <code>format</code> attributes and arguments (see the
 	documentation for <code>format</code> attributes for limitations).
 	At present, 8-bit character types are supported.</li>
-	<li>LTO-support.<br />Darwin has benefited from ongoing work on
-	LTO; support for this is now stable and enabled by default.</li>
 	<li>Object file size reduction.<br /> The Darwin zeroed memory
 	allocators have been re-written to make more use of
 	<code>.zerofill</code> sections.  For non-debug code, this can

Re: darwin LTO broken under Xcode 3.2.6/4.0

[Jack Howarth]

On Sun, Mar 13, 2011 at 09:38:06PM +0100, Steven Bosscher wrote:
> 
> I agree it is probably better to re-code things, but that will be
> impossible do before GCC 4.6 goes out.
> 
> We have to make a decision: keep the LTO support for Mach-O in for now
> and recommend a non-recent Xcode, disable it for the GCC 4.6 release
> and for the GCC 4.5.x series.
> 
> Ciao!
> Steven

Steven,
   I've come to the conclusion that your suggestion to disable lto for
gcc 4.6.0 on darwin is likely the correct course. The reason is that Apple
only provides Xcode 3.2.2 which I am pretty sure is missing some critical fixes
for http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43751 and others issues.
The users will be unable to obtain anything newer from Apple except Xcode 3.2.6
via Software Update which has the buggy assembler. Also any user currently
on Xcode 3.2.5 is subject to an accidentally upgrade to 3.2.6 via Software Update
if they are careless. It is probably better to disable the lto now rather than
suffer the complaints from end-users later. We can just revert...

2010-09-03  Jack Howarth <howarth@bromo.med.uc.edu>

        * configure.ac: Enable LTO by default on Darwin.
        * configure: Regenerate.

to achieve this.
                Jack