[Bug lto/53282] New: lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[Bug lto/53282] New: lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

             Bug #: 53282
           Summary: lto and visibility-inlines-hidden makes "wrongly"
                    hidden symbols and in a way that depends on the order
                    of the input compilation units
    Classification: Unclassified
           Product: gcc
           Version: 4.7.1
            Status: UNCONFIRMED
          Severity: blocker
          Priority: P3
         Component: lto
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: vincenzo.innocente@cern.ch
            Target: x86_64-unknown-linux-gnu
             Build: gcc-4_7-branch revision 187276


Created attachment 27345
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=27345
directory containing three (very similar) real-life file after pre-compiler
step

in the following example (not much reduced from a real-life code)
a symbol, very similar to others, is made hidden for no good reason.
Apparently it also depends on the order of the input files…


bzip2 -d ltobug3.tar.bz2 
tar -xf ltobug3.tar 
cd ltobug3/
c++ -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so d1.cc d2.cc
d3.cc ; nm -C bha.so | grep "makeKey" | grep -v "type"
0000000000004290 t
_ZN3edm10eventsetup15heterocontainer7makeKeyI11DummyRecordNS0_19EventSetupRecordKeyEEET0_v.local.77.4195
0000000000004750 W edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<Dummy2Record,
edm::eventsetup::EventSetupRecordKey>()
00000000000048a0 W edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<DepRecord,
edm::eventsetup::EventSetupRecordKey>()

c++ -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so d2.cc d1.cc
d3.cc ; nm -C bha.so | grep "makeKey" | grep -v "type"
00000000000048a0 W edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<DummyRecord,
edm::eventsetup::EventSetupRecordKey>()
0000000000004750 W edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<Dummy2Record,
edm::eventsetup::EventSetupRecordKey>()
00000000000049f0 W edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<DepRecord,
edm::eventsetup::EventSetupRecordKey>()

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #1 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-08 13:51:34 UTC ---
This has nothing to do with -fvisbility-inlines-hidden (well, that might
be necessary to trigger it).  LTO brings symbols local to ship them to
multiple partitions and partition layout (unfortunately, but probably
also unavoidably) depends on linker file order.

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #2 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-08 14:01:27 UTC ---
understood why it is related to the order of the input files.
Still that particular symbol shall not be hidden.
I've a library with 307 of those symbols, and lto hides only 7 of them!

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org

--- Comment #3 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-05-08 16:35:57 UTC ---
LTO works in a way that it first collect whole program and promote all symbols
that are not externally visible static.  After doing whole program optimization
the program is decomposed again into ltrans unit and compiled independently.
Symbols used from multiple ltrans unit are made hidden, so the units can refer
to it.

In your program apparently only those few symbols are crossing boundary of
ltrans. So this is really independent thing.

You can try to compile with -flto-partition=none.  It will take longer but
should avoid the hidden symbols.

File order independency of partitioning is possible, but I doubt it is
desirable design goal. In fact keeping things approximately in the order it was
passed to linker makes sense unless compiler knows better.

Honza

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #4 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 06:29:40 UTC ---
Hi Honza,
I forgot to say that I tried both 
-flto-partition=1to1
and
-flto-partition=none
with the same result

the point is that the symbols in question refers to a templated function
explicitly defined as
template  edm::eventsetup::EventSetupRecordKey
edm::eventsetup::heterocontainer::makeKey<_recordclassname_,
edm::eventsetup::EventSetupRecordKey>()

in the attached there are three of those: only one is hidden.
In the "real" library there are 307 of those, only 7 are hidden!
SO I suspect some sort of bug in lto

I will try to produce a synthetic test

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #5 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 09:10:35 UTC ---
(In reply to comment #4)
> Hi Honza,
> I forgot to say that I tried both 
> -flto-partition=1to1
> and
> -flto-partition=none
> with the same result
> 
> the point is that the symbols in question refers to a templated function
> explicitly defined as
> template  edm::eventsetup::EventSetupRecordKey
> edm::eventsetup::heterocontainer::makeKey<_recordclassname_,
> edm::eventsetup::EventSetupRecordKey>()
> 
> in the attached there are three of those: only one is hidden.
> In the "real" library there are 307 of those, only 7 are hidden!
> SO I suspect some sort of bug in lto
> 
> I will try to produce a synthetic test

All hidden symbols are postfixed with something like .local.77.4195, making
them no longer the symbols for the explicitely instantiated functions.

I suppose you say that the bug is that you have explicitely instantiated
some templates which you want to have appear in the library as exported
but LTO makes all instances hidden so that no externally visible symbol
for that explicitely instantiated template remains?

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #6 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 09:39:54 UTC ---
On 9 May, 2012, at 11:10 AM, rguenth at gcc dot gnu.org wrote:

> All hidden symbols are postfixed with something like .local.77.4195, making
> them no longer the symbols for the explicitely instantiated functions.
> 
> I suppose you say that the bug is that you have explicitely instantiated
> some templates which you want to have appear in the library as exported
> but LTO makes all instances hidden so that no externally visible symbol
> for that explicitely instantiated template remains?

correct.
The additional point is that this happens only for some symbols, not all, even
if the definitions/declarations are identical.

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

Richard Guenther <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2012-05-09
     Ever Confirmed|0                           |1

--- Comment #7 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 09:58:09 UTC ---
(In reply to comment #6)
> On 9 May, 2012, at 11:10 AM, rguenth at gcc dot gnu.org wrote:
> 
> > All hidden symbols are postfixed with something like .local.77.4195, making
> > them no longer the symbols for the explicitely instantiated functions.
> > 
> > I suppose you say that the bug is that you have explicitely instantiated
> > some templates which you want to have appear in the library as exported
> > but LTO makes all instances hidden so that no externally visible symbol
> > for that explicitely instantiated template remains?
> 
> correct.
> The additional point is that this happens only for some symbols, not all, even
> if the definitions/declarations are identical.

Ok, so the question would be - why does GCC think this symbol is not
possibly referenced from outside of the LTO unit?

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #8 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 10:03:33 UTC ---
On 9 May, 2012, at 11:58 AM, rguenth at gcc dot gnu.org wrote:

> Ok, so the question would be - why does GCC think this symbol is not
> possibly referenced from outside of the LTO unit?
Indeed.

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #9 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 10:15:21 UTC ---
Btw, the testcase does not reproduce for me.  What binutils version and which
linker do you use?  Do you actually end up using -fuse-linker-plugin?  Because
it reproduces for me with -fno-use-linker-plugin only (a path that is probably
not very well tested these days ...)

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #10 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 10:40:08 UTC ---
On 9 May, 2012, at 12:15 PM, rguenth at gcc dot gnu.org wrote:

> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282
> 
> --- Comment #9 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 10:15:21 UTC ---
> Btw, the testcase does not reproduce for me.  What binutils version and which
> linker do you use?  Do you actually end up using -fuse-linker-plugin?  Because
> it reproduces for me with -fno-use-linker-plugin only (a path that is probably
> not very well tested these days …)we use
ld -v
GNU gold (GNU Binutils 2.21.1) 1.11

and we do not use explicitely -fuse-linker-plugin,

gcc is configured as follow
c++ -v
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/afs/cern.ch/user/i/innocent/w3/gcc47slc5/libexec/gcc/x86_64-unknown-linux-gnu/4.7.1/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ./configure --enable-languages=c,c++,fortran
--disable-multilib --enable-gold=yes --disable-nls --enable-lto
--with-mpc=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--with-gmp=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--with-mpfr=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--prefix=/afs/cern.ch/user/i/innocent/w3/gcc47slc5 : (reconfigured) ./configure
--enable-languages=c,c++,fortran --disable-multilib --enable-gold=yes
--disable-nls --enable-lto
--with-mpc=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--with-gmp=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--with-mpfr=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0
--prefix=/afs/cern.ch/user/i/innocent/w3/gcc47slc5
Thread model: posix
gcc version 4.7.1 20120508 (prerelease) [gcc-4_7-branch revision 187276] (GCC) 

>

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #11 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 10:47:02 UTC ---
if I add -fuse-linker-plugin I get
c++: error: -fuse-linker-plugin is not supported in this configuration

my understanding was that since 4.6 linker-plugin was not required anymore for
lto

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #12 from Jan Hubicka <hubicka at ucw dot cz> 2012-05-09 12:21:38 UTC ---
> if I add -fuse-linker-plugin I get
> c++: error: -fuse-linker-plugin is not supported in this configuration
> 
> my understanding was that since 4.6 linker-plugin was not required anymore for
> lto

No, LTO plugin is needed for "proper" LTO and will always be.  We have
alternative
non-plugin path that results in poorer code quality.

In non-plugin path there is logic to privatize comdats whose address is not
taken, since they can be duplicated.  With explicit instantiation this code
however should not trigger.  I will check.

Honza

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #13 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 12:35:51 UTC ---
On 9 May, 2012, at 2:21 PM, hubicka at ucw dot cz wrote:

> No, LTO plugin is needed for "proper" LTO and will always be.  We have
> alternative
> non-plugin path that results in poorer code quality.
I see. 

could you please specify the configure options to be used ?
I tried 
--enable-linker-plugin
and does not help

thois is how I configured latest version of the trunk
c++ -v
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/afs/cern.ch/user/i/innocent/w2/libexec/gcc/x86_64-unknown-linux-gnu/4.8.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ./configure --prefix=/afs/cern.ch/user/i/innocent/w2
--enable-languages=c,c++,fortran -enable-gold=yes --enable-lto
--with-build-config=bootstrap-lto --with-gmp-lib=/usr/local/lib64
--with-mpfr-lib=/usr/local/lib64 -with-mpc-lib=/usr/local/lib64
--enable-cloog-backend=isl --with-cloog=/usr/local
--with-ppl-lib=/usr/local/lib64 CFLAGS='-O2 -ftree-vectorize -fPIC'
CXXFLAGS='-O2 -fPIC -ftree-vectorize -fvisibility-inlines-hidden -march=native'
-enable-libitm -disable-multilib --enable-linker-plugin
Thread model: posix
gcc version 4.8.0 20120509 (experimental) [trunk revision 187326] (GCC) 

and I get
 c++ -fuse-linker-plugin -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o
bha.so d1.cc d2.cc d3.cc
c++: error: -fuse-linker-plugin is not supported in this configuration

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #14 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 13:45:15 UTC ---
(In reply to comment #11)
> if I add -fuse-linker-plugin I get
> c++: error: -fuse-linker-plugin is not supported in this configuration
> 
> my understanding was that since 4.6 linker-plugin was not required anymore for
> lto

Since GCC 4.6 a linker-plugin is used automatically if a suitable linker is
found.  If that doesn't work for you that means the configure check

"linker plugin support"

fails - you have to check why.  Did you build binutils with support for
plugins?

[Bug lto/53282] lto and visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #15 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-09 14:48:20 UTC ---
I'm indeed using a "ld" that is not the one of the system.

googling around I found this:
http://comments.gmane.org/gmane.comp.gcc.help/41304
titled" g++: error: -fuse-linker-plugin is not supported in this configuration"
at some point it hints to the fact the gcc configure seems to ignore the ld in
the path
and uses the system one, detects no plugin etc.


I configured gcc with --with-ld=/usr/local/bin/ld and it does not help
LD=/usr/local/bin/ld  neither.

I'm sure that plugin is supported by this version of ld.
I wil try more, any suggestion is appreciated.


in the mean time  we can change the bug report saying that it happens if the
linker-plugin is not used…

[Bug lto/53282] [lto w/o linker plugin] visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

vincenzo Innocente <vincenzo.innocente at cern dot ch> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|lto and                     |[lto w/o linker plugin]
                   |visibility-inlines-hidden   |visibility-inlines-hidden
                   |makes "wrongly" hidden      |makes "wrongly" hidden
                   |symbols and in a way that   |symbols and in a way that
                   |depends on the order of the |depends on the order of the
                   |input compilation units     |input compilation units
           Severity|blocker                     |minor

[Bug lto/53282] [lto w/o linker plugin] visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #16 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-10 09:45:47 UTC ---
(In reply to comment #15)
> I'm indeed using a "ld" that is not the one of the system.
> 
> googling around I found this:
> http://comments.gmane.org/gmane.comp.gcc.help/41304
> titled" g++: error: -fuse-linker-plugin is not supported in this configuration"
> at some point it hints to the fact the gcc configure seems to ignore the ld in
> the path
> and uses the system one, detects no plugin etc.
> 
> 
> I configured gcc with --with-ld=/usr/local/bin/ld and it does not help
> LD=/usr/local/bin/ld  neither.
> 
> I'm sure that plugin is supported by this version of ld.
> I wil try more, any suggestion is appreciated.
> 
> 
> in the mean time  we can change the bug report saying that it happens if the
> linker-plugin is not used…

I think you need to use --with-plugin-ld.  Looking at how the "default" linker
is determined it seems you would need to set DEFAULT_LINKER, at least I cannot
see how --with-ld would end up in $gcc_cv_ld (configure would need to print
your supplied ld when looking for ld and for the ld to use for the target).

[Bug lto/53282] [lto w/o linker plugin] visibility-inlines-hidden makes "wrongly" hidden symbols and in a way that depends on the order of the input compilation units

[vincenzo.innocente at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282

--- Comment #17 from vincenzo Innocente <vincenzo.innocente at cern dot ch> 2012-05-10 09:55:42 UTC ---
I managed to get the linker-plugin enabled
1) building gcc in a directory different than the svn source one
2) using
--with-build-time-tools=
to point where we have all "current" tool-set

now I see
grep "linker plugin " build.log 
checking linker plugin support... 2
checking linker plugin support... 2
checking linker plugin support... 2

when previously was … 0

now we are fighting to install the new binutils above an old distro…

thanks for all the support given so far