[LTO merge][0/15] Description of the final 15 patches

[LTO merge][0/15] Description of the final 15 patches

[Diego Novillo]

I will be sending the final 15 patches to bring all the
functionality from the LTO branch.  I tried to divide the patches
along maintainer lines, but there are some overlaps

- C FE
	Adds processing of -flto and -fwhopr.

- C++ FE
	Adds a langhook used by need_assembler_name_p to handle
	templates.  This implements Jason's suggestion from
	http://gcc.gnu.org/ml/gcc-patches/2009-09/msg00534.html

- Fortran FE
	Adds processing of -flto and -fwhopr.  Note that the
	Fortran testsuite still shows ~140 failures with LTO
	enabled.  IIUC, most of these have corresponding bugs
	filed and they are not LTO-related (in that LTO exposes
	some bugs in the FE).  We should discuss whether we want
	to include Fortran support for LTO in 4.5.

- Java FE
	There is no LTO support for Java, but since LTO uses
	zlib, Java does not need to include it.

- LTO FE
	This is the new front end for GIMPLE.

- Configury
	Detection of libelf, LTO support, etc.

- Contrib
	A minor patch to add lto-plugin to the timestamps.

- Documentation
	Documentation for the different LTO options.

- Driver
	The somewhat ugly changes to collect2 and the main gcc
	driver to support spawning lto1 when we detect a .o file
	with LTO sections in it.

- Streamer
	The code needed to support the on-disk representation of
	GIMPLE, CFG and cgraph.

- Langhooks
	New langhooks to support manipulating ELF sections and
	the new langhook for asking the FE if a DECL needs to
	have an assembler name set.

- libiberty
	I need help with this one.  When the linker plugin is
	enabled (if GCC is configured to use gold), LTO can
	detect LTO objects inside archives via the callbacks it
	gets from the linker.  Since the linker plugin is a
	shared object, and it uses libiberty functions, it needs
	to use a shared libiberty.

	Currently, we just force --enable-shared on libiberty,
	but I would only want to do that if gold and lto are
	enabled.  We detect gold and lto support in the top
	configure script, but how do I send that down to
	libiberty's configure?

- Linker plugin
	This is the set of callbacks that gold uses to talk with
	lto1 and resolve symbols out of object files in the link.

- Middle end
	All sorts of changes, mostly in the callgraph, IPA and
	pass manager.

- Testsuite
	New tests for LTO and new dejagnu support for writing LTO
	tests.  Most of this has already been reviewed by Janis.


Major outstanding work:

- Support for debugging information.  Currently, -flto -g does
  not work (no debug information is generated).  This is because
  free_lang_data does not preserve enough information for the
  debug hooks to emit it.  I will fix this during stage 3.

- Outstanding bugzilla (http://tinyurl.com/yarz5q9).

- Some 'FIXME lto' markers.  These are cleanup markers, several
  related to WHOPR.  We could probably debate whether to include
  the WHOPR functionality in 4.5.  It certainly is not very
  different from LTO, but it does need some TLC.

Once I get all the approvals I need, I will request trunk to be
frozen for a day or so to give me time to commit the whole thing.
I already have a merged local tree and this exact same code is in
the LTO branch.


Thanks.  Diego.

Re: [LTO merge][0/15] Description of the final 15 patches

[Joseph S. Myers]

On Mon, 28 Sep 2009, Diego Novillo wrote:

> - libiberty
> 	I need help with this one.  When the linker plugin is
> 	enabled (if GCC is configured to use gold), LTO can
> 	detect LTO objects inside archives via the callbacks it
> 	gets from the linker.  Since the linker plugin is a
> 	shared object, and it uses libiberty functions, it needs
> 	to use a shared libiberty.
> 
> 	Currently, we just force --enable-shared on libiberty,
> 	but I would only want to do that if gold and lto are
> 	enabled.  We detect gold and lto support in the top
> 	configure script, but how do I send that down to
> 	libiberty's configure?

Shared libiberty seems like a bad idea as you then need to deal with 
soname allocation and changing the soname whenever an ABI-incompatible 
change is made.  What you actually need is a PIC libiberty to link into 
the plugin (and avoiding using any global data in libiberty that needs a 
single copy in any program; hopefully it doesn't have any), not a shared 
one.

(You also need to make sure this doesn't affect what is built for the 
target, only for the host.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [LTO merge][0/15] Description of the final 15 patches

[Ian Lance Taylor]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

"Joseph S. Myers" <joseph@codesourcery.com> writes:

> On Mon, 28 Sep 2009, Diego Novillo wrote:
>
>> - libiberty
>> 	I need help with this one.  When the linker plugin is
>> 	enabled (if GCC is configured to use gold), LTO can
>> 	detect LTO objects inside archives via the callbacks it
>> 	gets from the linker.  Since the linker plugin is a
>> 	shared object, and it uses libiberty functions, it needs
>> 	to use a shared libiberty.
>> 
>> 	Currently, we just force --enable-shared on libiberty,
>> 	but I would only want to do that if gold and lto are
>> 	enabled.  We detect gold and lto support in the top
>> 	configure script, but how do I send that down to
>> 	libiberty's configure?
>
> Shared libiberty seems like a bad idea as you then need to deal with 
> soname allocation and changing the soname whenever an ABI-incompatible 
> change is made.  What you actually need is a PIC libiberty to link into 
> the plugin (and avoiding using any global data in libiberty that needs a 
> single copy in any program; hopefully it doesn't have any), not a shared 
> one.

Yes.  Fortunately there is no such thing as a shared libiberty.  If
you configure libiberty with --enable-shared, what you get is a PIC
libiberty (libiberty/pic/libiberty.a) alongside the non-PIC one
(libiberty/libiberty.a).

So all Diego needs to do is pass --enable-shared down to libiberty
when --enable-lto/--enable-gold.  The way to do that is something like
the appended.

Ian


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: shared libiberty --]
[-- Type: text/x-diff, Size: 1144 bytes --]

Index: configure.ac
===================================================================
--- configure.ac	(revision 152253)
+++ configure.ac	(working copy)
@@ -2495,6 +2495,11 @@ case $enable_bootstrap in
     ;;
 esac
 
+extra_host_libiberty_configure_flags=
+if test "$enable_gold" = "yes" -a "$enable_lto" = "yes"; then
+  extra_host_libiberty_configure_flags=--enable-shared
+fi
+
 AC_MSG_CHECKING(for default BUILD_CONFIG)
 
 AC_ARG_WITH([build-config],
Index: Makefile.def
===================================================================
--- Makefile.def	(revision 152253)
+++ Makefile.def	(working copy)
@@ -97,7 +97,8 @@ host_modules= { module= ld; bootstrap=tr
 host_modules= { module= libcpp; bootstrap=true; };
 host_modules= { module= libdecnumber; bootstrap=true; };
 host_modules= { module= libgui; };
-host_modules= { module= libiberty; bootstrap=true; };
+host_modules= { module= libiberty; bootstrap=true;
+	        extra_configure_flags='@extra_host_libiberty_configure_flags@'};
 // We abuse missing to avoid installing anything for libiconv.
 host_modules= { module= libiconv;
 		extra_configure_flags='--disable-shared';

Re: [LTO merge][0/15] Description of the final 15 patches

[Diego Novillo]

On Tue, Sep 29, 2009 at 10:47, Ian Lance Taylor <iant@google.com> wrote:

> So all Diego needs to do is pass --enable-shared down to libiberty
> when --enable-lto/--enable-gold.  The way to do that is something like
> the appended.

Thanks.  Testing.


Diego.

Re: [LTO merge][0/15] Description of the final 15 patches

[Paolo Bonzini]

> So all Diego needs to do is pass --enable-shared down to libiberty
> when --enable-lto/--enable-gold.  The way to do that is something like
> the appended.

What about just always adding --enable-shared to the host libiberty?

Paolo

Re: [LTO merge][0/15] Description of the final 15 patches

[Ian Lance Taylor]

Paolo Bonzini <bonzini@gnu.org> writes:

>> So all Diego needs to do is pass --enable-shared down to libiberty
>> when --enable-lto/--enable-gold.  The way to do that is something like
>> the appended.
>
> What about just always adding --enable-shared to the host libiberty?

That will just cause everybody to always compile libiberty twice.  Why
do that if we don't have to?

Ian

Re: [LTO merge][0/15] Description of the final 15 patches

[Paolo Bonzini]

On 09/29/2009 06:52 PM, Ian Lance Taylor wrote:
> Paolo Bonzini<bonzini@gnu.org>  writes:
>
>>> So all Diego needs to do is pass --enable-shared down to libiberty
>>> when --enable-lto/--enable-gold.  The way to do that is something like
>>> the appended.
>>
>> What about just always adding --enable-shared to the host libiberty?
>
> That will just cause everybody to always compile libiberty twice.  Why
> do that if we don't have to?

It's just a matter of time before we need to do that, anyway (say, when 
we have plugins that we distribute by default, for example for mudflap). 
  Conditionalizing on GCC could be a middle ground, or maybe the worst 
possible choice? :-)

Paolo

Re: [LTO merge][0/15] Description of the final 15 patches

[Ian Lance Taylor]

Paolo Bonzini <bonzini@gnu.org> writes:

> On 09/29/2009 06:52 PM, Ian Lance Taylor wrote:
>> Paolo Bonzini<bonzini@gnu.org>  writes:
>>
>>>> So all Diego needs to do is pass --enable-shared down to libiberty
>>>> when --enable-lto/--enable-gold.  The way to do that is something like
>>>> the appended.
>>>
>>> What about just always adding --enable-shared to the host libiberty?
>>
>> That will just cause everybody to always compile libiberty twice.  Why
>> do that if we don't have to?
>
> It's just a matter of time before we need to do that, anyway (say,
> when we have plugins that we distribute by default, for example for
> mudflap). Conditionalizing on GCC could be a middle ground, or maybe
> the worst possible choice? :-)

The shared libiberty is only required for the gold plugin.  It's
possible to get benefits from LTO without the gold plugin.  Which is a
good thing, since gold only works for a few targets.

Ian

Re: [LTO merge][0/15] Description of the final 15 patches

[Joseph S. Myers]

On Mon, 28 Sep 2009, Diego Novillo wrote:

> I will be sending the final 15 patches to bring all the
> functionality from the LTO branch.  I tried to divide the patches
> along maintainer lines, but there are some overlaps

I'll go through the individual patches later, but a general comment:

You say "tested on x86_64".  More detail is needed.  Have you tested both 
LTO-enabled and LTO-disabled configurations, making sure there are no 
regressions in each case?  I think such testing of both configurations is 
needed.  In addition, an LTO-enabled cross from a non-ELF host to an ELF 
target should be tested (there are five non-ELF hosts in the secondary 
platforms list for 4.5; any one of them could be used for this, or a 
non-ELF host not on that list; I realise gold's plugin support may not yet 
be portable to such hosts so that part of the patch may not be testable 
for this case).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [LTO merge][0/15] Description of the final 15 patches

[Diego Novillo]

On Mon, Sep 28, 2009 at 22:00, Joseph S. Myers <joseph@codesourcery.com> wrote:

> You say "tested on x86_64".  More detail is needed.  Have you tested both

Sorry, I completely spaced out and forgot to mention this.  Various
folks are testing the branch.  I sent a request for testing the branch
(http://gcc.gnu.org/ml/gcc/2009-09/msg00566.html).  I'm using that
thread to keep track of breakage that should be fixed before the final
merge.

> LTO-enabled and LTO-disabled configurations, making sure there are no
> regressions in each case?

Yes.  Targets with no ELF support refuse to enable LTO and error out
at configure time if --enable-lto is forced.  There was a bug in some
of the testsuite patches that Richi fixed.


Diego.