Link with mixed IR/non-IR objects

Link with mixed IR/non-IR objects

[H.J.]

Here is a proposal to link with mixed IR/non-IR objects:

* 2 kinds of object files
  o non-IR object file has
    * non-IR sections
  o IR object file has
    * IR sections
    * non-IR sections
    * The output of "ld -r" with mixed IR/non-IR objects should work
with:
        o Compilers/linkers with IR support.
	o Compilers/linkers without IR support.
* Add the mixed object file which has
  o IR sections
  o non-IR sections:
    * Object codes from IR sections.
    * Object codes from non-IR object files.
  o Object-only section:
    * With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
type
    on ELF:
    #define SHT_GNU_OBJECT_ONLY 0x6ffffff8	/* Object only */
    * Contain non-IR object file.
    * Input is discarded after link.
* Linker action:
  o Classify each input object file:
    * If there is a ".gnu_object_only" section, it is a mixed object
file.
    * If there is a IR section, it is an IR object file.
    * Otherwise, it is a non-IR object file.
  o Relocatable non-IR link:
    * Prepare for an object-only output.
    * Prepare for a regular output.
    * For each mixed object file:
      * Add IR and non-IR sections to the regular output.
      * For object-only section:
	* Extract object only file.
	* Add it to the object-only output.
	* Discard object-only section.
    * For each IR object file:
      * Add IR and non-IR sections to the regular output.
    * For each non-IR object file:
      * Add non-IR sections to the regular output.
      * Add non-IR sections to the object-only output.
    * Final output:
      * If there are IR objects, non-IR objects and the object-only
      output isn't empty:
	* Put the object-only output into the object-only section.
	* Add the object-only section to the regular output.
	* Remove the object-only output.
  o Normal link and relocatable IR link:
    * Prepare for output.
    * IR link:
      * For each mixed object file:
	* Compile and add IR sections to the output.
	* Discard non-IR sections.
	* Object-only section:
	  * Extract object only file.
	  * Add it to the output.
	  * Discard object-only section.
      * For each IR object file:
        * Compile and add IR sections to the output.
	* Discard non-IR sections.
      * For each non-IR object file:
	* Add non-IR sections to the output.
    * Non-IR link:
      * For each mixed object file:
	* Add non-IR sections to the output.
	* Discard IR sections and object-only section.
      * For each IR object file:
	* Add non-IR sections to the output.
	* Discard IR sections.
      * For each non-IR object file:
	* Add non-IR sections to the output.

Re: Link with mixed IR/non-IR objects

[Markus Trippelsdorf]

On 2011.04.27 at 06:49 -0700, H.J. wrote:
> Here is a proposal to link with mixed IR/non-IR objects:
> 
> * 2 kinds of object files
>   o non-IR object file has
>     * non-IR sections
>   o IR object file has
>     * IR sections
>     * non-IR sections
>     * The output of "ld -r" with mixed IR/non-IR objects should work
> with:
>         o Compilers/linkers with IR support.
> 	o Compilers/linkers without IR support.
> * Add the mixed object file which has
>   o IR sections
>   o non-IR sections:
>     * Object codes from IR sections.
>     * Object codes from non-IR object files.
>   o Object-only section:
>     * With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
> type
>     on ELF:
>     #define SHT_GNU_OBJECT_ONLY 0x6ffffff8	/* Object only */

I'm wondering if it wouldn't be better to simply use:
SHT_GNU_IRBITS (= 0x6fff4952)
that has already been proposed here:
http://gcc.gnu.org/wiki/whopr/driver ?

»Thus, we plan to mark the IR sections with a special section type,
SHT_GNU_IRBITS (= 0x6fff4952).«

-- 
Markus

Re: Link with mixed IR/non-IR objects

[H.J. Lu]

On Wed, Apr 27, 2011 at 7:29 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2011.04.27 at 06:49 -0700, H.J. wrote:
>> Here is a proposal to link with mixed IR/non-IR objects:
>>
>> * 2 kinds of object files
>>   o non-IR object file has
>>     * non-IR sections
>>   o IR object file has
>>     * IR sections
>>     * non-IR sections
>>     * The output of "ld -r" with mixed IR/non-IR objects should work
>> with:
>>         o Compilers/linkers with IR support.
>>       o Compilers/linkers without IR support.
>> * Add the mixed object file which has
>>   o IR sections
>>   o non-IR sections:
>>     * Object codes from IR sections.
>>     * Object codes from non-IR object files.
>>   o Object-only section:
>>     * With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
>> type
>>     on ELF:
>>     #define SHT_GNU_OBJECT_ONLY 0x6ffffff8    /* Object only */
>
> I'm wondering if it wouldn't be better to simply use:
> SHT_GNU_IRBITS (= 0x6fff4952)
> that has already been proposed here:
> http://gcc.gnu.org/wiki/whopr/driver ?
>
> »Thus, we plan to mark the IR sections with a special section type,
> SHT_GNU_IRBITS (= 0x6fff4952).«
>

They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
any IR. It encapsulates a relocatable object file without any IR bits.


-- 
H.J.

Re: Link with mixed IR/non-IR objects

[Markus Trippelsdorf]

On 2011.04.27 at 07:38 -0700, H.J. Lu wrote:
> On Wed, Apr 27, 2011 at 7:29 AM, Markus Trippelsdorf
> <markus@trippelsdorf.de> wrote:
> > On 2011.04.27 at 06:49 -0700, H.J. wrote:
> >> Here is a proposal to link with mixed IR/non-IR objects:
> >>
> >> * 2 kinds of object files
> >>   o non-IR object file has
> >>     * non-IR sections
> >>   o IR object file has
> >>     * IR sections
> >>     * non-IR sections
> >>     * The output of "ld -r" with mixed IR/non-IR objects should work
> >> with:
> >>         o Compilers/linkers with IR support.
> >>       o Compilers/linkers without IR support.
> >> * Add the mixed object file which has
> >>   o IR sections
> >>   o non-IR sections:
> >>     * Object codes from IR sections.
> >>     * Object codes from non-IR object files.
> >>   o Object-only section:
> >>     * With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
> >> type
> >>     on ELF:
> >>     #define SHT_GNU_OBJECT_ONLY 0x6ffffff8    /* Object only */
> >
> > I'm wondering if it wouldn't be better to simply use:
> > SHT_GNU_IRBITS (= 0x6fff4952)
> > that has already been proposed here:
> > http://gcc.gnu.org/wiki/whopr/driver ?
> >
> > »Thus, we plan to mark the IR sections with a special section type,
> > SHT_GNU_IRBITS (= 0x6fff4952).«
> >
> 
> They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
> any IR. It encapsulates a relocatable object file without any IR bits.

Of course. What I was trying to ask was if SHT_GNU_IRBITS and
SHT_GNU_OBJECT_ONLY aren't exactly complementary to each other?

(IOW, if SHT_GNU_IRBITS would be implemented, would there still be a
need for SHT_GNU_OBJECT_ONLY?)
-- 
Markus

Re: Link with mixed IR/non-IR objects

[H.J. Lu]

On Wed, Apr 27, 2011 at 8:06 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2011.04.27 at 07:38 -0700, H.J. Lu wrote:
>> On Wed, Apr 27, 2011 at 7:29 AM, Markus Trippelsdorf
>> <markus@trippelsdorf.de> wrote:
>> > On 2011.04.27 at 06:49 -0700, H.J. wrote:
>> >> Here is a proposal to link with mixed IR/non-IR objects:
>> >>
>> >> * 2 kinds of object files
>> >>   o non-IR object file has
>> >>     * non-IR sections
>> >>   o IR object file has
>> >>     * IR sections
>> >>     * non-IR sections
>> >>     * The output of "ld -r" with mixed IR/non-IR objects should work
>> >> with:
>> >>         o Compilers/linkers with IR support.
>> >>       o Compilers/linkers without IR support.
>> >> * Add the mixed object file which has
>> >>   o IR sections
>> >>   o non-IR sections:
>> >>     * Object codes from IR sections.
>> >>     * Object codes from non-IR object files.
>> >>   o Object-only section:
>> >>     * With section name ".gnu_object_only" and SHT_GNU_OBJECT_ONLY
>> >> type
>> >>     on ELF:
>> >>     #define SHT_GNU_OBJECT_ONLY 0x6ffffff8    /* Object only */
>> >
>> > I'm wondering if it wouldn't be better to simply use:
>> > SHT_GNU_IRBITS (= 0x6fff4952)
>> > that has already been proposed here:
>> > http://gcc.gnu.org/wiki/whopr/driver ?
>> >
>> > »Thus, we plan to mark the IR sections with a special section type,
>> > SHT_GNU_IRBITS (= 0x6fff4952).«
>> >
>>
>> They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
>> any IR. It encapsulates a relocatable object file without any IR bits.
>
> Of course. What I was trying to ask was if SHT_GNU_IRBITS and
> SHT_GNU_OBJECT_ONLY aren't exactly complementary to each other?

They are independent of each other.

> (IOW, if SHT_GNU_IRBITS would be implemented, would there still be a
> need for SHT_GNU_OBJECT_ONLY?)

We sill need SHT_GNU_OBJECT_ONLY to support mixing IR/non-IR
object files.

-- 
H.J.

Re: Link with mixed IR/non-IR objects

[Alan Modra]

On Wed, Apr 27, 2011 at 07:38:23AM -0700, H.J. Lu wrote:
> They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
> any IR. It encapsulates a relocatable object file without any IR bits.

I think we first should ask whether GNU ld needs to support ld -r
with mixed IR and normal object files.  I'm inclined to think that ld
should not do the segregation of object files and effectively perform
two linking operations with one output file stored inside the other.
Why can't we simply issue an error if someone tries to mix object
files like this?

-- 
Alan Modra
Australia Development Lab, IBM

Re: Link with mixed IR/non-IR objects

[H.J. Lu]

On Wed, Apr 27, 2011 at 5:23 PM, Alan Modra <amodra@gmail.com> wrote:
> On Wed, Apr 27, 2011 at 07:38:23AM -0700, H.J. Lu wrote:
>> They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
>> any IR. It encapsulates a relocatable object file without any IR bits.
>
> I think we first should ask whether GNU ld needs to support ld -r
> with mixed IR and normal object files.  I'm inclined to think that ld
> should not do the segregation of object files and effectively perform
> two linking operations with one output file stored inside the other.
> Why can't we simply issue an error if someone tries to mix object
> files like this?
>

This is for usability. Many projects use "ld -r" on object files
generated from C/C++ sources, assembly codes as well as
from the third party vendors.  If "ld -r" doesn't work on mixed IR
and non-IR object files, they have to change the build infrastructure
if they want to use LTO.

-- 
H.J.

Re: Link with mixed IR/non-IR objects

[Alan Modra]

On Wed, Apr 27, 2011 at 07:06:44PM -0700, H.J. Lu wrote:
> On Wed, Apr 27, 2011 at 5:23 PM, Alan Modra <amodra@gmail.com> wrote:
> > On Wed, Apr 27, 2011 at 07:38:23AM -0700, H.J. Lu wrote:
> >> They are different. SHT_GNU_OBJECT_ONLY section doesn't contain
> >> any IR. It encapsulates a relocatable object file without any IR bits.
> >
> > I think we first should ask whether GNU ld needs to support ld -r
> > with mixed IR and normal object files.  I'm inclined to think that ld
> > should not do the segregation of object files and effectively perform
> > two linking operations with one output file stored inside the other.
> > Why can't we simply issue an error if someone tries to mix object
> > files like this?
> >
> 
> This is for usability. Many projects use "ld -r" on object files
> generated from C/C++ sources, assembly codes as well as
> from the third party vendors.  If "ld -r" doesn't work on mixed IR
> and non-IR object files, they have to change the build infrastructure
> if they want to use LTO.

IMNSHO it is a mistake to use ld -r as a means of packaging object
files.  Projects that do so ought to be encouraged to move away from
that practice, because ld -r doesn't just simply package object
files.  It can move code around, losing cache locality.  You can also
lose out on possible final link optimisations, eg. --gc-sections,
powerpc64 multi-toc.  Thin archives, or even @ response files, are a
better solution.

So I think we'd be doing the world a service by *not* supporting ld -r
with mixed objects.  I also don't like the added complexity in the
linker, and in the object file format.  Third party tools won't handle
these mixed object files.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Link with mixed IR/non-IR objects

[Cary Coutant]

> IMNSHO it is a mistake to use ld -r as a means of packaging object
> files.  Projects that do so ought to be encouraged to move away from
> that practice, because ld -r doesn't just simply package object
> files.  It can move code around, losing cache locality.  You can also
> lose out on possible final link optimisations, eg. --gc-sections,
> powerpc64 multi-toc.  Thin archives, or even @ response files, are a
> better solution.
>
> So I think we'd be doing the world a service by *not* supporting ld -r
> with mixed objects.  I also don't like the added complexity in the
> linker, and in the object file format.  Third party tools won't handle
> these mixed object files.

I'm also not much in favor of using ld -r as a means for packaging
files, but it still has other uses. Those other uses typically involve
combining modules and hiding symbols, types of activities that might
also go well with actually doing localized LTO during the link, and ld
-r ought to work fine in that scenario.

Nonetheless, I think it can be useful to run ld -r while deferring LTO
to the final link, but my problem with this proposal is the whole
concept of nesting an object file inside a section of another object
file. We already have at least two mechanisms for this kind of
encapsulation: archive libraries on the one hand, and section groups
on the other. I think either one of those could be used to solve the
problem without inventing a third kind of encapsulation.

I suggested earlier a scheme where the linker could generate an
archive of objects, one member being the result of an ld -r on all the
non-IR inputs, the other(s) being the IR inputs. A trivial extension
to the archive format could designate to the linker that all members
must be loaded.

For something more like the object-inside-a-section solution, I don't
see why section groups couldn't be made to work. ELF has a group type
that currently defines only the one type of group (COMDAT). You could
define a new type of group that would contain the mixed compiled-code
and IR sections from a mixed-IR input file, and the result of an ld -r
over a mixed input would contain regular sections representing the
combination of the non-IR inputs, and a group of sections for each IR
input.

Either one of these alternatives sounds better to me than embedding a
whole object in a section inside another object file.

-cary

Re: Link with mixed IR/non-IR objects

[Cary Coutant]

Sigh, the generic-abi mailing list rejected my reply when I sent it
from my work address.


---------- Forwarded message ----------
From: Cary Coutant <ccoutant@google.com>
Date: Fri, Apr 29, 2011 at 3:35 PM
Subject: Re: Link with mixed IR/non-IR objects
To: generic-abi@googlegroups.com, binutils@sourceware.org


> IMNSHO it is a mistake to use ld -r as a means of packaging object
> files.  Projects that do so ought to be encouraged to move away from
> that practice, because ld -r doesn't just simply package object
> files.  It can move code around, losing cache locality.  You can also
> lose out on possible final link optimisations, eg. --gc-sections,
> powerpc64 multi-toc.  Thin archives, or even @ response files, are a
> better solution.
>
> So I think we'd be doing the world a service by *not* supporting ld -r
> with mixed objects.  I also don't like the added complexity in the
> linker, and in the object file format.  Third party tools won't handle
> these mixed object files.

I'm also not much in favor of using ld -r as a means for packaging
files, but it still has other uses. Those other uses typically involve
combining modules and hiding symbols, types of activities that might
also go well with actually doing localized LTO during the link, and ld
-r ought to work fine in that scenario.

Nonetheless, I think it can be useful to run ld -r while deferring LTO
to the final link, but my problem with this proposal is the whole
concept of nesting an object file inside a section of another object
file. We already have at least two mechanisms for this kind of
encapsulation: archive libraries on the one hand, and section groups
on the other. I think either one of those could be used to solve the
problem without inventing a third kind of encapsulation.

I suggested earlier a scheme where the linker could generate an
archive of objects, one member being the result of an ld -r on all the
non-IR inputs, the other(s) being the IR inputs. A trivial extension
to the archive format could designate to the linker that all members
must be loaded.

For something more like the object-inside-a-section solution, I don't
see why section groups couldn't be made to work. ELF has a group type
that currently defines only the one type of group (COMDAT). You could
define a new type of group that would contain the mixed compiled-code
and IR sections from a mixed-IR input file, and the result of an ld -r
over a mixed input would contain regular sections representing the
combination of the non-IR inputs, and a group of sections for each IR
input.

Either one of these alternatives sounds better to me than embedding a
whole object in a section inside another object file.

-cary

Re: Link with mixed IR/non-IR objects

[Alan Modra]

On Fri, Apr 29, 2011 at 03:38:50PM -0700, Cary Coutant wrote:
> I suggested earlier a scheme where the linker could generate an
> archive of objects, one member being the result of an ld -r on all the
> non-IR inputs, the other(s) being the IR inputs. A trivial extension
> to the archive format could designate to the linker that all members
> must be loaded.

Yes, and all of this except for the extension to the archive format
could be done in a wrapper (perhaps even the gcc driver).

> For something more like the object-inside-a-section solution, I don't
> see why section groups couldn't be made to work. ELF has a group type
> that currently defines only the one type of group (COMDAT). You could
> define a new type of group that would contain the mixed compiled-code
> and IR sections from a mixed-IR input file, and the result of an ld -r
> over a mixed input would contain regular sections representing the
> combination of the non-IR inputs, and a group of sections for each IR
> input.

Won't work without using separate symbol tables or an extension to
mark symbols as IR-only.  The linker should detect when a symbol is
only defined in IR files and report an error if the symbol is then
referenced in a relocation.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Link with mixed IR/non-IR objects

[Cary Coutant]

>> For something more like the object-inside-a-section solution, I don't
>> see why section groups couldn't be made to work. ELF has a group type
>> that currently defines only the one type of group (COMDAT). You could
>> define a new type of group that would contain the mixed compiled-code
>> and IR sections from a mixed-IR input file, and the result of an ld -r
>> over a mixed input would contain regular sections representing the
>> combination of the non-IR inputs, and a group of sections for each IR
>> input.
>
> Won't work without using separate symbol tables or an extension to
> mark symbols as IR-only.  The linker should detect when a symbol is
> only defined in IR files and report an error if the symbol is then
> referenced in a relocation.

No extension is necessary -- if the symbol is defined in an IR input,
the st_shndx field will point to a member of an IR section group.
There shouldn't be a problem with relocations that reference such a
symbol -- it would be just like a non-IR object that references a
symbol defined in an IR file. Just like COMDAT groups, it's OK to
reference a global symbol from outside the group, but not OK to
reference a local symbol from outside.

In this model, the plugin would tell the linker that it's claiming the
IR section groups but leaving everything else as if it was a normal
object file to be linked. The linker would then ignore the IR groups,
treating symbol definitions in those groups as UNDEFs, with any
relocations to those symbols being references from outside the IR
world.

For complete consistency, IR files generated by the compiler should
put all the sections inside an IR group. Then we'd have a consistent
model -- IR-only files would have just IR sections in an IR group;
"fat" files would have both IR and compiled sections in an IR group;
and "mixed" (ld -r) files would have multiple IR groups plus regular
compiled sections not in an IR group.

IR groups could be IR-only or fat. When running without a plugin, the
linker would process fat groups as regular sections (ignoring the IR
sections like it does today), and would need to give an error if it
finds an IR-only group (don't remember if we're smart enough to do
that today with an IR-only file).

-cary

Re: Link with mixed IR/non-IR objects

[Alan Modra]

On Mon, May 02, 2011 at 10:51:36PM -0700, Cary Coutant wrote:
> >> For something more like the object-inside-a-section solution, I don't
> >> see why section groups couldn't be made to work. ELF has a group type
> >> that currently defines only the one type of group (COMDAT). You could
> >> define a new type of group that would contain the mixed compiled-code
> >> and IR sections from a mixed-IR input file, and the result of an ld -r
> >> over a mixed input would contain regular sections representing the
> >> combination of the non-IR inputs, and a group of sections for each IR
> >> input.
> >
> > Won't work without using separate symbol tables or an extension to
> > mark symbols as IR-only.  The linker should detect when a symbol is
> > only defined in IR files and report an error if the symbol is then
> > referenced in a relocation.
> 
> No extension is necessary -- if the symbol is defined in an IR input,
> the st_shndx field will point to a member of an IR section group.

That's true for most symbols but not true for common or absolute
symbols, or any symbols using the OS or PROC specific sections.  If
you don't treat SHN_COMMON IR-only symbols as special, the linker will
needlessly allocate space for them.

And, yes, I wasn't correct to say that the linker should report errors
on relocations against IR-only symbols.  As you said, the linker
should treat these as undefined.

-- 
Alan Modra
Australia Development Lab, IBM