ln -r and cherry picking.

ln -r and cherry picking.

[Kenneth Zadeck]

I want to point out that the current implementation of lto is not
compatible with "ln -r", and will need to be modified to support
"cherry picking" the function bodies.

In the current implementation, each lto section (such as what holds
a function body or the streamed information from an ipa pass)
references an index that is unique to that .o file.  This index allows
the encoding of the function body or the ipa information to use small
integers to reference the global types and declarations.

"ln -r" will not work in the current system for two reasons:

  1) The ipa passes currently just read the ipa sections until they
  hit their "end of file marker".  Assuming that we create these
  section with attributes to tell "ln -r" to concatenate them (I have
  no idea how to do this but I assume it is easy), the ipa pass's
  stream readers will need to modified to restart after the end of
  file marker and do this until the number of bytes in the section is
  exhausted.  This should only require wrapping them in an extra loop.

  2) LTO sections need to be able to find "their index" of decls and
  types.  By "their index" I mean the index that each section used to
  reference the decls and types when the section was generated.

  There is currently no identifying fingerprint in a .o file to match
  an index with the sections that need it.  Note that the indexes for
  different .o files are (obviously) different.  We will need to add a
  fingerprint for each .o file so that when sections from different .o
  files are merged, we can match a section with the proper index.  I
  currently do not know how to generate a finger print.  I assume some
  string with the name of the machine, the process id and the time is
  what we want, but the right answer could also be something like the
  md5 checksum of some large part of the .o file.   

(2) is also an issue for the cherry picker to deal with, because the
picker is not only going to have to cherry pick but it is going to
have to regenerate all of the indexes based on how the decls and types
are merged.

Note that having this index external to the section that holds a
function body is the only way for the cherry picking to work without
having the function bodies modified.  If explicit references to the
decls and types were in the function bodies, you would have to rewrite
the function bodies to make them point to the proper merged type or
decl.

I do not want to tackle either of these tasks until the streaming lto
branch is merged into the lto branch, but this is certainly something
that will need to be done.

Kenny

Re: ln -r and cherry picking.

[Arnaud Charlet]

> I want to point out that the current implementation of lto is not
> compatible with "ln -r", and will need to be modified to support
> "cherry picking" the function bodies.

I assume you mean "ld -r", right ?

Arno

Re: ln -r and cherry picking.

[Kenneth Zadeck]

Arnaud Charlet wrote:
>> I want to point out that the current implementation of lto is not
>> compatible with "ln -r", and will need to be modified to support
>> "cherry picking" the function bodies.
>>     
>
> I assume you mean "ld -r", right ?
>
> Arno
>   
yes, of course. Dennis Richie's curse: two letter commands.

Re: ln -r and cherry picking.

[Cary Coutant]

>  2) LTO sections need to be able to find "their index" of decls and
>  types.  By "their index" I mean the index that each section used to
>  reference the decls and types when the section was generated.

Can't you just put an ELF symbol (can be an unnamed local -- could
even just be a section symbol) on the index section, then add a
pointer in the IR section with a relocation to that symbol? This is
basically how DWARF .debug_info sections point to the abbrev table in
the .debug_abbrev sections.

-cary

Re: ln -r and cherry picking.

[Kenneth Zadeck]

Cary Coutant wrote:
>>  2) LTO sections need to be able to find "their index" of decls and
>>  types.  By "their index" I mean the index that each section used to
>>  reference the decls and types when the section was generated.
>>     
>
> Can't you just put an ELF symbol (can be an unnamed local -- could
> even just be a section symbol) on the index section, then add a
> pointer in the IR section with a relocation to that symbol? This is
> basically how DWARF .debug_info sections point to the abbrev table in
> the .debug_abbrev sections.
>
> -cary
>   
I think that one of the goals here is to not make that too dependent on 
elf.  For instance, we are in the process of getting rid of all of the 
dwarf.  After maddox does that, our only dependence on elf will be as a 
container to hold all of the sections.  

Given that gcc is not always an elf compiler, it really is a lot easier 
to invent your own wheel for something like this rather that using elf's 
wheel for the first target and then having to figure out how make 
someone else wheel fit the for the rest of the targets.

kenny

Re: ln -r and cherry picking.

[Cary Coutant]

> I think that one of the goals here is to not make that too dependent on elf.
>  For instance, we are in the process of getting rid of all of the dwarf.
>  After maddox does that, our only dependence on elf will be as a container
> to hold all of the sections.
> Given that gcc is not always an elf compiler, it really is a lot easier to
> invent your own wheel for something like this rather that using elf's wheel
> for the first target and then having to figure out how make someone else
> wheel fit the for the rest of the targets.

This is basic functionality that *every* object file format supports.
I don't think using a symbol and a relocation is going to tie you down
to ELF -- no more so than the idea of using sections to store your
data in.

I also think it's simpler and more deterministic than using a hash.

-cary