[Bug lto/47247] New: Linker plugin specification makes it difficult to handle COMDATs

[Bug lto/47247] New: Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

           Summary: Linker plugin specification makes it difficult to
                    handle COMDATs
           Product: gcc
           Version: 4.6.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: hubicka@gcc.gnu.org


Current linker plugin specification has problem with COMDAT.  When COMDAT
fnuction is put into symbol table and is used, it gets resolved as PREVAILING.
This imply that it must be output in the final binary.

This is often suboptimal because inlining often eliminate need for the out of
line copy of COMDAT. GCC works around by not putting COMDAT symbols into symbol
table. This leads to posibility that both LTO and non-LTO objects will define
variant of the COMDAT function and that GCC will bring the LTO one static and
thus two copies of COMDAT will be present. For this reason GCC will not do this
for COMDAT function that is not virtual and has address taken.

This trick works well in most cases, but leads i.e. to undefined symbols when
Mozilla is built with LTO and -fprofile-generate
https://bugzilla.mozilla.org/show_bug.cgi?id=624385

this is because GCC no longer optimize out some functions from unused header.
This is because indirect call profiling takes their address and forces them to
stay in the program.

In longer therm we really need better solution for this.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2011.01.10 17:00:06
     Ever Confirmed|0                           |1

--- Comment #1 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-01-10 17:00:06 UTC ---
Testcase to demonstrate the problem
jh@evans:~> cat t2.C
#include <stdlib.h>
extern int unused2;
extern int unused (void);
class test
{
public:
  virtual int unused3()
  {
    return (int)(size_t)&unused;
  }
};
extern inline int unused (void)
{
  test t;
  if (t.unused3())
    return unused2;
}
main()
{
}
jh@evans:~> /abuild/jh/trunk-install/bin/g++ -O2 t2.C  -rdynamic
jh@evans:~> /abuild/jh/trunk-install/bin/g++ -O2 t2.C  -rdynamic -flto
/abuild/jh/trunk-install/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/../../../../x86_64-unknown-linux-gnu/bin/ld:
/tmp/ccvAd49o.ltrans0.ltrans.o: in function
unused():ccvAd49o.ltrans0.o(.text._Z6unusedv+0xc): error: undefined reference
to 'unused2'
collect2: ld returned 1 exit status

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #2 from H.J. Lu <hjl.tools at gmail dot com> 2011-01-10 17:02:02 UTC ---
Do we have a small testcase to experiment with?

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> 2011-01-10 17:04:30 UTC ---
Can we mark the symbol COMDAT when we generate the output?

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #4 from Jan Hubicka <hubicka at ucw dot cz> 2011-01-10 17:10:52 UTC ---
> Can we mark the symbol COMDAT when we generate the output?
What symbol and what output?

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> 2011-01-10 17:13:46 UTC ---
(In reply to comment #4)
> > Can we mark the symbol COMDAT when we generate the output?
> What symbol and what output?
> 
> Honza

I don't think hjl/lto-mixed branch has the problem

---
Current linker plugin specification has problem with COMDAT.  When COMDAT
fnuction is put into symbol table and is used, it gets resolved as PREVAILING.
This imply that it must be output in the final binary.
---

if all COMAT symbols are properly marked since PREVAILING is
only used for symbol resolution and has no impact on symbol
table.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #6 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-01-10 17:54:51 UTC ---
The plugin specification says that once the COMDAT is marked PREVAILING, it has
to be output.
  "Any symbol marked PREVAILING_DEF must be defined in one object file added to
the link after WPA is done, or an undefined symbol error will result. Any
symbol marked PREVAILING_DEF_IRONLY may be left undefined (provided all
references to the symbol have been removed), and the linker will not issue an
error."

GCC will optimize it out only if it is set PREVAILING_IRONLY, but I think we
conlcuded earlier that this can not be used for symbols visible to dynamic
linking (and GCC would use this info to bring the COMDAT symbol static). 

So I am convinced that with current API it is impossible and we need something
like PREVAILING_IRONLY_EXPORTED or so for the case of symbol that is not used
in current DSO but is exported. GCC can then use the knowledge of what COMDATs
can be taken away.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #7 from H.J. Lu <hjl.tools at gmail dot com> 2011-01-10 18:26:00 UTC ---
(In reply to comment #6)
> The plugin specification says that once the COMDAT is marked PREVAILING, it has
> to be output.
>   "Any symbol marked PREVAILING_DEF must be defined in one object file added to
> the link after WPA is done, or an undefined symbol error will result. Any
> symbol marked PREVAILING_DEF_IRONLY may be left undefined (provided all
> references to the symbol have been removed), and the linker will not issue an
> error."
> 
> GCC will optimize it out only if it is set PREVAILING_IRONLY, but I think we
> conlcuded earlier that this can not be used for symbols visible to dynamic
> linking (and GCC would use this info to bring the COMDAT symbol static). 
> 

What undesirable things may happen if we mark a COMDAT symbol
PREVAILING_DEF? Is that we won't know which one will be used
if both LTO and non-LTO objects define the same COMAT symbol?

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[ccoutant at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

Cary Coutant <ccoutant at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ccoutant at gcc dot gnu.org

--- Comment #8 from Cary Coutant <ccoutant at gcc dot gnu.org> 2011-01-10 18:50:37 UTC ---
The plugin interface needs LDPR_PREVAILING_DEF_IRONLY_BUT_EXPORTED (but with a
shorter name). For discussion, see:

http://gcc.gnu.org/ml/gcc/2010-12/msg00035.html
http://gcc.gnu.org/ml/gcc/2010-12/msg00043.html
http://gcc.gnu.org/ml/gcc/2010-12/msg00045.html

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[ccoutant at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #9 from Cary Coutant <ccoutant at gcc dot gnu.org> 2011-01-10 19:07:54 UTC ---
I've added a new disposition code LDPR_PREVAILING_DEF_IRONLY_EXP and a new
version of the GET_SYMBOLS interface to the API specification on the wiki:

http://gcc.gnu.org/wiki/whopr/driver

How does that look?

-cary

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #10 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-01-10 20:56:23 UTC ---
> What undesirable things may happen if we mark a COMDAT symbol
> PREVAILING_DEF? Is that we won't know which one will be used
> if both LTO and non-LTO objects define the same COMAT symbol?

excatly, we don't know if something else in the same DSO is defining this
COMDAT.
Also the plugin specs explicitely say that plugin is required to provide this
symbol in the output per the paragraph I quoted.

Cary: the update seems fine to me. Any chance to get it implemented on plugin
side? Adding support in GCC is easy (and we can even keep the current trick of
not outputing some COMDATs in place until old linkers gets out of busyness)

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #11 from H.J. Lu <hjl.tools at gmail dot com> 2011-01-10 21:01:55 UTC ---
(In reply to comment #10)
> > What undesirable things may happen if we mark a COMDAT symbol
> > PREVAILING_DEF? Is that we won't know which one will be used
> > if both LTO and non-LTO objects define the same COMAT symbol?
> 
> excatly, we don't know if something else in the same DSO is defining this
> COMDAT.
> Also the plugin specs explicitely say that plugin is required to provide this
> symbol in the output per the paragraph I quoted.
> 
> Cary: the update seems fine to me. Any chance to get it implemented on plugin
> side? Adding support in GCC is easy (and we can even keep the current trick of
> not outputing some COMDATs in place until old linkers gets out of busyness)

Adding to hjl/lto-mixed branch should be easy. If you can send
me a GCC patch, I will update hjl/lto-mixed branch.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rafael.espindola at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #12 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-14 19:59:22 UTC ---
A quick summary to see if got this right:

Currently the linker has three options for a symbol in a comdat:

*) pick one not in the IL. The plugin gets a PREEMPTED_REG
*) pick one in the IL and give PREVAILING_DEF to the plugin
*) as above, but PREVAILING_DEF_IRONLY

The first one is undesirable since the more LTO sees, the better.
The second one prevents gcc from dropping tho symbol if it drops the uses.
To give the third option the plugin would have to know the semantics of this
symbol (available anywhere it is used).

The proposal is that PREVAILING_DEF_IRONLY_EXP will let gcc drop all references
if it drops all uses.

Some cases:

*) a visible symbol in IL and there is a reference from it in a ELF file. Gold
must give PREVAILING_DEF to the plugin and it cannot drop it.
*) a visible symbol in IL and there is no references from it in a ELF file.
Gold gives PREVAILING_DEF_IRONLY_EXP to the plugin. What gcc can do then
depends on what the symbol is
  1) if the symbol is one (like vtable) known to be available where it is
needed, gcc can drop it if it drops all uses
  2) if the symbol is a simple weak symbol gcc must keep it
*) a local symbol. gold gives PREVAILING_DEF_IRONLY to the plugin and it can do
anything it wants with it.

On LLVM land I think this translates to

*) If given PREVAILING_DEF, linkonce gets upgrade to weak and linkonce_odr to
weak_odr
*) If given PREVAILING_DEF_IRONLY_EXP, likages stay as they are
*) If given PREVAILING_DEF_IRONLY the symbol gets internalized

I think this should work, thanks.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #13 from Jan Hubicka <hubicka at ucw dot cz> 2011-02-15 18:49:15 UTC ---
> A quick summary to see if got this right:

Yes, the summary seems right.

Note that GCC plays now games with not putting COMDATs into LTO symbol table
unless they really need to be unique.  So if you have simple inline functions
with no address taken things the function is never put into LTO symbol table
and linker doesn't care. GCC then merge the different copies itself and promote
the function into static function.  This worlsf well except for possibility of
having two instances of the function (one in LTO world that is promoted to
static function and one in non-LTO code staying as comdat).
This works well for Firefox ;))

Things gets ugly with variables and stuff that has address taken. This is less
common in C++ code but still worrysome.

> On LLVM land I think this translates to
> 
> *) If given PREVAILING_DEF, linkonce gets upgrade to weak and linkonce_odr to
> weak_odr
> *) If given PREVAILING_DEF_IRONLY_EXP, likages stay as they are
> *) If given PREVAILING_DEF_IRONLY the symbol gets internalized
> 
> I think this should work, thanks.

I am not sure if you are allowed to change COMDAT into weak. The plugin spec
explicitely speaks of comdat symbol staying and I am not sure if Gold/GNU ld
are ready for symbol changing their type. 

Taking PREVAILING_DEF is silently accepted by both gold and GNU ld, but in gold
it results in stale dynamic symbol table entries.  GCC originally behaved on
PREVAILING_DEF as if it was PREVAILING_DEF_IRONLY_EXP and it did result in
massive dynamic symbol table section increases.

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rafael.espindola at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #14 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-15 19:39:09 UTC ---
Sorry, can you expand on what gcc was doing that was causing it to expand the
dynamic symbol table?

With LLVM what we are doing is

*) Report all symbols
*) Any symbol not given a PREVAILING_DEF, we make it internal.
*) All other symbols stay the same.

For example, a vtable is normally produce as a symbol with linkonce_odr (see
http://llvm.org/docs/LangRef.html#linkage_linkonce_odr). If it is not used, we
drop it.

The potential problem with what LLVM is doing (I don't have a seem it in
practice, but haven't tried to force it) is the case

* file a.cc has the vtable for class foo and a use of it
* file b.cc has the vtable for class foo and a use of it

We produce IL for a.cc and a ELF for b.cc. Gold decides to use the IL version,
gives the plugin a PREVAILING_DEF. LLVM optimises the use away and drops the
vtable. We now have an undefined reference to the vtable.

With both PREVAILING_DEF_IRONLY_EXP and PREVAILING_DEF llvm would be able to
tell the difference. If there was no use of the vtable from an ELF file, gold
would give a PREVAILING_DEF_IRONLY_EXP and llvm would be allowed to drop it. If
there is, gold gives a PREVAILING_DEF and llvm must upgrade the vtable from
linkonce_odr to weak_odr which causes it to say in the final .o even if it
optimises out all uses.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #15 from Jan Hubicka <hubicka at ucw dot cz> 2011-02-15 23:20:40 UTC ---
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247
> 
> --- Comment #14 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-15 19:39:09 UTC ---
> Sorry, can you expand on what gcc was doing that was causing it to expand the
> dynamic symbol table?
> 
> With LLVM what we are doing is
> 
> *) Report all symbols
> *) Any symbol not given a PREVAILING_DEF, we make it internal.
> *) All other symbols stay the same.
> 
> For example, a vtable is normally produce as a symbol with linkonce_odr (see

The problem is with dropping linkonce_odr that has been previously reported.
This way gold will allocate entry in the dynamic symbol table (you can see it
in
nm of the final binary) with no definition/use.
Once something is given PREVAILING_DEF, it can not be optimized away.

> We produce IL for a.cc and a ELF for b.cc. Gold decides to use the IL version,
> gives the plugin a PREVAILING_DEF. LLVM optimises the use away and drops the
> vtable. We now have an undefined reference to the vtable.

Yes, that is problem, too, but I didn't see it in practice.  
> 
> With both PREVAILING_DEF_IRONLY_EXP and PREVAILING_DEF llvm would be able to
> tell the difference. If there was no use of the vtable from an ELF file, gold
> would give a PREVAILING_DEF_IRONLY_EXP and llvm would be allowed to drop it. If
> there is, gold gives a PREVAILING_DEF and llvm must upgrade the vtable from
> linkonce_odr to weak_odr which causes it to say in the final .o even if it
> optimises out all uses.

Yes, that seems right except that GCC makes difference in between comdat
symbols that must stay even if they are optimized out and weak symbols since
both are handled bit differently.  I am not sure if updating comdat symbol to
weak is safe in ELF world, probably Iant would know.

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rafael.espindola at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #16 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-16 04:03:36 UTC ---
> The problem is with dropping linkonce_odr that has been previously reported.
> This way gold will allocate entry in the dynamic symbol table (you can see it
> in
> nm of the final binary) with no definition/use.
> Once something is given PREVAILING_DEF, it can not be optimized away.

I see. Even with PREVAILING_DEF_IRONLY_EXP we still have to update gold to drop
those, no? Gold doesn't know the language semantics to know which visible
symbols can or cannot be dropped, so it can only assume the plugin knows what
it is doing when it drops one.

> > We produce IL for a.cc and a ELF for b.cc. Gold decides to use the IL version,
> > gives the plugin a PREVAILING_DEF. LLVM optimises the use away and drops the
> > vtable. We now have an undefined reference to the vtable.
> 
> Yes, that is problem, too, but I didn't see it in practice.

Yes, the common case is to try to put as much as possible in the IL :-)
Now that I can build firefox in LTO I will try to create a testcase for this
next week.

> > 
> > With both PREVAILING_DEF_IRONLY_EXP and PREVAILING_DEF llvm would be able to
> > tell the difference. If there was no use of the vtable from an ELF file, gold
> > would give a PREVAILING_DEF_IRONLY_EXP and llvm would be allowed to drop it. If
> > there is, gold gives a PREVAILING_DEF and llvm must upgrade the vtable from
> > linkonce_odr to weak_odr which causes it to say in the final .o even if it
> > optimises out all uses.
> 
> Yes, that seems right except that GCC makes difference in between comdat
> symbols that must stay even if they are optimized out and weak symbols since
> both are handled bit differently.  I am not sure if updating comdat symbol to
> weak is safe in ELF world, probably Iant would know.

Well, since we know this is the symbol being used, llvm can just upgrade it all
the way to a regular symbol.

> Honza

Thanks,
Rafael

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #17 from Jan Hubicka <hubicka at ucw dot cz> 2011-02-16 07:50:48 UTC ---
> I see. Even with PREVAILING_DEF_IRONLY_EXP we still have to update gold to drop
> those, no? Gold doesn't know the language semantics to know which visible

I assume it will be part of PREVAILING_DEF_IRONLY_EXP implementation.

> symbols can or cannot be dropped, so it can only assume the plugin knows what
> it is doing when it drops one.

Yes.
> 
> Well, since we know this is the symbol being used, llvm can just upgrade it all
> the way to a regular symbol.

Definitly it has to stay weak or things will break when multiple shared
libraries
export the same previously COMDAT symbol.

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rafael.espindola at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #18 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-16 16:14:52 UTC ---
I have created a "small" test of the symbol table problem. It is in

http://people.mozilla.com/~respindola/test.tar.xz

The test is firefox's libxul with most files copied into one directory to make
it easy to run. The test script runs the link twice. First with the IL files
and then with combined .o file.

The sizes are

39339456 libxul1.so
34436696 libxul2.so

The output of "size" in both files is


For a 5 MB reduction I might end up writing a wrapper that runs ld twice :-)

23218068    2380480     289668    25888216    18b05d8    libxul1.so
23175418    2380480     289668    25845566    18a5f3e    libxul2.so

Re: [Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[Jan Hubicka]

> 39339456 libxul1.so
> 34436696 libxul2.so

Yep, it seems similar to what I was getting.  Quite important difference and all the stuff
gets loaded into memory at startup by dynamic linker.

> For a 5 MB reduction I might end up writing a wrapper that runs ld twice :-)

You might try GNU-ld's plugin support (I am not sure if it has same problem,
but I think it doesn't, definitly not with HJ's double linking branch).

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #19 from Jan Hubicka <hubicka at ucw dot cz> 2011-02-16 23:12:32 UTC ---
> 39339456 libxul1.so
> 34436696 libxul2.so

Yep, it seems similar to what I was getting.  Quite important difference and
all the stuff
gets loaded into memory at startup by dynamic linker.

> For a 5 MB reduction I might end up writing a wrapper that runs ld twice :-)

You might try GNU-ld's plugin support (I am not sure if it has same problem,
but I think it doesn't, definitly not with HJ's double linking branch).

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[ccoutant at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #20 from ccoutant at google dot com 2011-02-16 23:35:28 UTC ---
> I have created a "small" test of the symbol table problem. It is in
>
> http://people.mozilla.com/~respindola/test.tar.xz
>
> The test is firefox's libxul with most files copied into one directory to make
> it easy to run. The test script runs the link twice. First with the IL files
> and then with combined .o file.
>
> The sizes are
>
> 39339456 libxul1.so
> 34436696 libxul2.so

> For a 5 MB reduction I might end up writing a wrapper that runs ld twice :-)

That 5MB difference is all due to the dynamic symbol table? I think we
can fix that in gold. : )

-cary

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rafael.espindola at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #21 from Rafael Avila de Espindola <rafael.espindola at gmail dot com> 2011-02-17 01:13:35 UTC ---
Most of it is in the string table. Ian gave me some pointers and I will try to
fix it tomorrow :-)

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #22 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-08-27 17:35:27 UTC ---
Carry,
is there any chance to move ahead on this problem?  I see you posted the
PREVAILING_DEF_IRONLY_EXP but it was never committed.

Concerning Rafael's comment:

 Why is PREVAILING_DEF_IRONLY_EXP needed? I think it can be solved with 
just PREVAILING_DEF if we say that the compiler can drop references to 
them if it knows they are provided in any DSO. The only case I can see 
where this would almost be a problem is something like

This is not possible with current formulation of plugin API. It explicitely
says "Any symbol marked PREVAILING_DEF must be defined in one object file added
to the link after WPA is done, or an undefined symbol error will result"

We could also go that route I guess, but
 1) We will have to bump API version or something to indicate whether compiler
is allowed to drop the symbols or not.  With current binutils implementation
bad things happens when compiler decided to drop the symbol it is not supposed
to (my original implementation did so and it resulted in problems, especially
in large dynamic linker tables)
 2) the LTO produced COMDAT function will likely be better, so we will lose bit
of code quality
 3) There are cases where function is output COMDAT but it is keyed (i.e. with
the repo files). In this case we will not be able to drop the symbol when it
becomes unnecessary at LTO time because we will not know whether the symbol is
used by non-LTO code or not.

All those reasons are rather side cases and thus weak, but because we are
effectively changing API anyway and because PREVAILING_DEF_IRONLY_EXP makes
compiler a bit more informed in well defined way, I would go for it.

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[ccoutant at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

Cary Coutant <ccoutant at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
         AssignedTo|unassigned at gcc dot       |ccoutant at gcc dot gnu.org
                   |gnu.org                     |

--- Comment #23 from Cary Coutant <ccoutant at gcc dot gnu.org> 2011-09-26 20:42:20 UTC ---
Gold patch posted:

http://sourceware.org/ml/binutils/2011-09/msg00191.html

-cary

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[ccoutant at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #24 from Cary Coutant <ccoutant at gcc dot gnu.org> 2011-09-26 23:32:17 UTC ---
Author: ccoutant
Date: Mon Sep 26 23:32:13 2011
New Revision: 179220

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=179220
Log:
    PR lto/47247
    * plugin-api.h (enum ld_plugin_symbol_resolution): Add
    LDPR_PREVAILING_DEF_IRONLY_EXP.
    (enum ld_plugin_tag): Add LDPT_GET_SYMBOLS_V2.

Modified:
    trunk/include/ChangeLog
    trunk/include/plugin-api.h

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #25 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-09-28 12:38:29 UTC ---
Thanks for gold support. GCC support is now posted at
http://gcc.gnu.org/ml/gcc-patches/2011-09/msg01818.html

We miss the GNU LD variant
http://sourceware.org/bugzilla/show_bug.cgi?id=13229

Dave, HJ, would be possible to look into that?

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #26 from Jan Hubicka <hubicka at gcc dot gnu.org> 2011-10-02 10:41:27 UTC ---
Author: hubicka
Date: Sun Oct  2 10:41:24 2011
New Revision: 179424

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=179424
Log:
    PR lto/47247
    * lto-plugin.c (get_symbols_v2): New variable.
    (write_resolution): Use V2 API when available.
    (onload): Handle LDPT_GET_SYMBOLS_V2.

    * lto-symtab.c (lto_symtab_resolve_symbols): Do not resolve
    when resolution is already availbale from plugin.
    (lto_symtab_merge_decls_1): Handle LDPR_PREVAILING_DEF_IRONLY_EXP.
    * cgraph.c (ld_plugin_symbol_resolution): Add prevailing_def_ironly_exp.
    * lto-cgraph.c (LDPR_NUM_KNOWN): Update.
    * ipa.c (varpool_externally_visible_p): IRONLY variables are never
    externally visible.
    * varasm.c (resolution_to_local_definition_p): Add
    LDPR_PREVAILING_DEF_IRONLY_EXP.
    (resolution_local_p): Likewise.

    * common.c (lto_resolution_str): Add new resolution.
    * common.h (lto_resolution_str): Likewise.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/cgraph.c
    trunk/gcc/ipa.c
    trunk/gcc/lto-cgraph.c
    trunk/gcc/lto-symtab.c
    trunk/gcc/lto/ChangeLog
    trunk/gcc/lto/common.c
    trunk/gcc/lto/common.h
    trunk/gcc/varasm.c
    trunk/lto-plugin/ChangeLog
    trunk/lto-plugin/lto-plugin.c

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

vincenzo Innocente <vincenzo.innocente at cern dot ch> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |vincenzo.innocente at cern
                   |                            |dot ch

--- Comment #27 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-10 07:14:51 UTC ---
is all this in 4.7?
with
GNU gold (GNU Binutils 2.21.1) 1.11
and
gcc-4_7-branch revision 187326

with linker-plugin enabled I get this type of errors
ld: error: hidden symbol
'_ZThn16_N5boost16exception_detail10clone_implINS0_19error_info_injectorISt11logic_errorEEED0Ev'
is not defined locally
ld: /tmp/innocent/ccggD26x.ltrans8.ltrans.o: in function
_ZTVN5boost16exception_detail10clone_implINS0_19error_info_injectorISt11logic_errorEEEE.local.4390:ccggD26x.ltrans8.o(.data.rel.ro+0x1558):
warning: relocation refers to discarded section
does not occur w/o linker-plugin

is the problem on gcc or gold side?
(posted also in http://sourceware.org/bugzilla/show_bug.cgi?id=12488)

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #28 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-10 07:35:02 UTC ---
I believe binutils 2.21.1 is not new enough, support for this was only added
on the 2.22 branch.

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #29 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-10 07:53:37 UTC ---
we will try binutil 2.22
in any case the test case is simple (with boost 1.49)
notice how w/o linker-plugin NO symbol at all are generated for 
boost::exception_detail::clone_impl etc.

cat boostLto.cc
#include "boost/shared_ptr.hpp"

struct A {
  A(){}
  A(int j) : i(j){}
int i;
};

boost::shared_ptr<A> make(int i) {
   return boost::shared_ptr<A>(new A(i));
}


rm bha.so ; c++ -fno-use-linker-plugin -fvisibility-inlines-hidden -O2 -flto
-shared -fPIC -o bha.so boostLto.cc
-I/afs/cern.ch/cms/slc5_amd64_gcc470/external/boost/1.49.0/include
-Wl,-z,defs,-E ; nm -C bha.so | grep clone
0000000000000ce0 t deregister_tm_clones
0000000000000d10 t register_tm_clones
[vocms123] /build/vin/CMSSW_6_0_X_2012-05-07-1500/src $ rm bha.so ; c++
-fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so boostLto.cc
-I/afs/cern.ch/cms/slc5_amd64_gcc470/external/boost/1.49.0/include
-Wl,-z,defs,-E ; nm -C bha.so | grep clone
                 U
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 U
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 U
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 w typeinfo for boost::exception_detail::clone_base
                 w typeinfo for
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>
                 w typeinfo name for boost::exception_detail::clone_base
                 w typeinfo name for
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>
                 U non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 U non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 U non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
                 U non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000001200 t deregister_tm_clones
0000000000001230 t register_tm_clones
[vocms123] /build/vin/CMSSW_6_0_X_2012-05-07-1500/src $ rm bha.so ; c++ -O2
-flto -shared -fPIC -o bha.so boostLto.cc
-I/afs/cern.ch/cms/slc5_amd64_gcc470/external/boost/1.49.0/include
-Wl,-z,defs,-E ; nm -C bha.so | grep clone0000000000001e00 t
_ZN5boost16exception_detail10clone_baseD0Ev.local.27.2724
0000000000001de0 t _ZN5boost16exception_detail10clone_baseD1Ev.local.40.2721
0000000000001de0 t _ZN5boost16exception_detail10clone_baseD2Ev.local.41.2807
0000000000002200 W
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000002170 W
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000002170 W
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000001f50 t
_ZNK5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_weak_ptrEEEE5cloneEv.local.14.2737
0000000000002000 t
_ZNK5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_weak_ptrEEEE7rethrowEv.local.16.2740
00000000000035a0 V typeinfo for boost::exception_detail::clone_base
00000000000035c0 V typeinfo for
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>
00000000000023c0 V typeinfo name for boost::exception_detail::clone_base
0000000000002400 V typeinfo name for
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>
0000000000003480 d _ZTVN5boost16exception_detail10clone_baseE.local.42.2718
00000000000033e0 d
_ZTVN5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_weak_ptrEEEEE.local.48.2728
0000000000002280 W non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
00000000000021f0 W non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000001ff0 t
_ZThn48_NK5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_weak_ptrEEEE5cloneEv.local.13.2747
00000000000020b0 t
_ZThn48_NK5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_weak_ptrEEEE7rethrowEv.local.15.2749
0000000000002270 W non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
00000000000021e0 W non-virtual thunk to
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
>::~clone_impl()
0000000000001c60 t deregister_tm_clones
0000000000001c90 t register_tm_clones

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #30 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-10 15:50:15 UTC ---
using the new binutil I've no more error
still warning of the type
tmp/innocent/ccmPaLCz.ltrans6.ltrans.o:ccmPaLCz.ltrans6.o:function
_ZTVN5boost16exception_detail10clone_implINS0_14bad_exception_EEE.local.4747:
warning: relocation refers to discarded section

any clue? 
any option to try?
not yet able to reproduce on a small example…

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #31 from Jan Hubicka <hubicka at ucw dot cz> 2012-05-10 17:07:47 UTC ---
> using the new binutil I've no more error
> still warning of the type
> tmp/innocent/ccmPaLCz.ltrans6.ltrans.o:ccmPaLCz.ltrans6.o:function
> _ZTVN5boost16exception_detail10clone_implINS0_14bad_exception_EEE.local.4747:
> warning: relocation refers to discarded section
> 
> any clue? 
> any option to try?
> not yet able to reproduce on a small example???
No idea.  It would be good to have a testcase.  If you fail to do so, could you
try to produce -fdump-ipa-cgraph dumps (some of them may land in TMP directory)
and send me those along with the output from linker and the resolution file
(you get
that one via --save-temps and it will be a file with .res extension. It may
have
funny name, usually it is -lm.res)?

Honza

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #32 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-10 17:13:20 UTC ---
On 10 May, 2012, at 7:07 PM, hubicka at ucw dot cz wrote:

> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247
> 
> --- Comment #31 from Jan Hubicka <hubicka at ucw dot cz> 2012-05-10 17:07:47 UTC ---
>> using the new binutil I've no more error
>> still warning of the type
>> tmp/innocent/ccmPaLCz.ltrans6.ltrans.o:ccmPaLCz.ltrans6.o:function
>> _ZTVN5boost16exception_detail10clone_implINS0_14bad_exception_EEE.local.4747:
>> warning: relocation refers to discarded section
>> 
>> any clue? 
>> any option to try?
>> not yet able to reproduce on a small example???
> No idea.  It would be good to have a testcase.  
found this
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644255
looks the very same warning
v.

> If you fail to do so, could you
> try to produce -fdump-ipa-cgraph dumps (some of them may land in TMP directory)
> and send me those along with the output from linker and the resolution file
> (you get
> that one via --save-temps and it will be a file with .res extension. It may
> have
> funny name, usually it is -lm.res)?
> 
> Honza
> 
> -- 
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.

--
Il est bon de suivre sa pente, pourvu que ce soit en montant. 
A.G.
http://www.flickr.com/photos/vin60/1320965757/

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #33 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-05-12 14:33:47 UTC ---
The actual problem was solved by V2 of plugin API.  The warning you describe
should not happen with or without the V2 API. Can you, please, try to fill in
independent PR possibly with a testcase?

[Bug lto/47247] Linker plugin specification makes it difficult to handle COMDATs

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47247

--- Comment #34 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-14 07:41:02 UTC ---
I've created PR53337