[Bug lto/42776] New: LTO doesn't work on non-ELF platforms.

[Bug lto/42776] New: LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

As per title, and see also the discussion in bug 41529.

There is no fundamental requirement for ELF to be the object format, as the
object file sections are just used as dumb containers for arbitrary binary
data.  The interface between lto-elf.c and the rest of lto1 needs only the
ability to create sections and to place and retrieve binary data in/from them,
so should be easy to implement for any object format that allows
arbitrarily-named sections; that's everything except a.out I would suppose.

I'm only planning to fix this for the PE-COFF (windows) platforms, but I'll try
and make the code general enough so that it works on other COFF platforms and
just needs enabling by the relevant target maintainer.


-- 
           Summary: LTO doesn't work on non-ELF platforms.
           Product: gcc
           Version: 4.5.0
            Status: UNCONFIRMED
          Keywords: build, lto
          Severity: normal
          Priority: P3
         Component: lto
        AssignedTo: davek at gcc dot gnu dot org
        ReportedBy: davek at gcc dot gnu dot org
 GCC build triplet: i686-pc-cygwin
  GCC host triplet: i686-pc-cygwin
GCC target triplet: i686-pc-cygwin


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #1 from davek at gcc dot gnu dot org  2010-01-17 16:09 -------
Created an attachment (id=19636)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19636&action=view)
work in progress patch

should be good for mingw as well.

needs some binutils support - will attach that here too, in case anyone wants
to try this patch out.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #2 from davek at gcc dot gnu dot org  2010-01-17 16:13 -------
Created an attachment (id=19637)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19637&action=view)
binutils support

This is needed to extend the .section directive in the pe-coff port of gas so
that we can tell it to byte-align the lto sections.  See Bug 40392 for why we
need the sections to be byte-aligned (actually just byte-size-granularitied,
but the way COFF works at the moment in binutils the two are linked) and Bug
40754 for why this would be bad on a risc platform (but it's ok on x86 where we
can do misaligned accesses.)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[d dot g dot gorbachev at gmail dot com]

------- Comment #3 from d dot g dot gorbachev at gmail dot com  2010-01-18 11:02 -------
(In reply to comment #1)

> work in progress patch

This seems to cause "*** No rule to make target `lto/@LTO_BINARY_READER@.o',
needed by `lto1'." error when build = host = target = i686-pc-linux-gnu


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #4 from davek at gcc dot gnu dot org  2010-01-18 11:05 -------
(In reply to comment #3)
> (In reply to comment #1)
> 
> > work in progress patch
> 
> This seems to cause "*** No rule to make target `lto/@LTO_BINARY_READER@.o',
> needed by `lto1'." error when build = host = target = i686-pc-linux-gnu

There's no diffs for the generated files in there; did you run autoconf?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[d dot g dot gorbachev at gmail dot com]

------- Comment #5 from d dot g dot gorbachev at gmail dot com  2010-01-18 11:51 -------
> did you run autoconf?

Forgot to run it, to my disgrace. :) Sorry, false alarm.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #6 from davek at gcc dot gnu dot org  2010-01-18 12:54 -------
(In reply to comment #5)
> > did you run autoconf?
> 
> Forgot to run it, to my disgrace. :) Sorry, false alarm.
> 

No need to apologise, thanks for testing on linux for me!


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[steven at gcc dot gnu dot org]

------- Comment #7 from steven at gcc dot gnu dot org  2010-01-18 16:18 -------
Should we perhaps rename all the lto_elf_ stuff to something else, if all of
this also Just Works with COFF?

Can we use a similar approach for Mach-O?

Big kudos for Dave, btw, for working on this.


-- 

steven at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
   Last reconfirmed|0000-00-00 00:00:00         |2010-01-18 16:18:01
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #8 from davek at gcc dot gnu dot org  2010-01-18 16:35 -------
(In reply to comment #7)
> Should we perhaps rename all the lto_elf_ stuff to something else, if all of
> this also Just Works with COFF?

  As I said, WIP; I was certainly thinking of renaming it all to
lto_objfile_xxx or some similar prefix in time for the final version.

> Can we use a similar approach for Mach-O?

  I don't speak Mach-O, but yes, the approach should work.  You'd start by
saying lto_binary_reader=lto-mach-o in config.gcc and adding a new
lto/lto-mach-o.c with the same handful of toplevel functions: open and close
file, build section hash, and create section and append binary data to section.

> Big kudos for Dave, btw, for working on this.

Ah, it's nothing - a simple COFF file reader is no BFD... (pun intended) ;-)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #9 from davek at gcc dot gnu dot org  2010-01-21 07:10 -------
Created an attachment (id=19672)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19672&action=view)
work in progress, revised to use unmodified binutils

D'oh.  Turns out p2align will do exactly what I want to the section alignment,
so there's no need to change binutils.  This is the resulting version that I'm
bootstrapping now.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #19636|0                           |1
        is obsolete|                            |
  Attachment #19637|0                           |1
        is obsolete|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #10 from davek at gcc dot gnu dot org  2010-01-21 07:14 -------
Created an attachment (id=19673)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19673&action=view)
Minor fix of previous attachment.

(In reply to comment #9)
>  This is the resulting version that I'm bootstrapping now.

Oops, that omitted the new files, so updated for your convenience.  Also, fixed
the "visilibity" typo.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #19672|0                           |1
        is obsolete|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #11 from davek at gcc dot gnu dot org  2010-01-21 19:46 -------
Today's take 2 produces a ton of decompression stream errors.  It turns out(*)
that the original approach probably is the correct one after all, and that
p2align will probably not do what we need here, so I've reinstated the original
patches.  

-- 
(*) - http://sourceware.org/ml/binutils/2010-01/msg00461.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[d dot g dot gorbachev at gmail dot com]

------- Comment #12 from d dot g dot gorbachev at gmail dot com  2010-01-22 01:17 -------
The patch works for mingw. So you can enable lto for it, too.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #13 from davek at gcc dot gnu dot org  2010-01-22 10:46 -------
(In reply to comment #12)
> The patch works for mingw. So you can enable lto for it, too.
> 

Thanks for that, I'll update the patch in the next day or three to include
MinGW.  (I'll also clean it up a bit and add more comments.)

Are you able to run the testsuite in the configuration you're building?  (i.e.
msys or cross-compile?)  If so, would you paste the output from running "make
check-gcc-c RUNTESTFLAGS=lto.exp" in the top level of your build dir please?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[d dot g dot gorbachev at gmail dot com]

------- Comment #14 from d dot g dot gorbachev at gmail dot com  2010-02-03 10:15 -------
There is a portability issue in is_elf_or_coff: fopen should be called with
"rb" instead of "r". Otherwise, fread fails when a COFF file has 26 sections,
because it is interpreted as a text, and 26 means "end of file".

(In reply to comment #13)

> Are you able to run the testsuite in the configuration you're building?  (i.e.
> msys or cross-compile?)  If so, would you paste the output from running "make
> check-gcc-c RUNTESTFLAGS=lto.exp" in the top level of your build dir please?

Not tried. GCC and Binutils can be built with -flto under MSYS, and they seem
to work.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #15 from davek at gcc dot gnu dot org  2010-02-03 12:02 -------
(In reply to comment #14)
> There is a portability issue in is_elf_or_coff: fopen should be called with
> "rb" instead of "r". Otherwise, fread fails when a COFF file has 26 sections,
> because it is interpreted as a text, and 26 means "end of file".

  ROFL, yes of course, good catch.  I should have remembered, there was just a
thread on the -patches list about adding "b" to the ELF LTO reader for the sake
of mingw-based cross-compilers!

> Not tried. GCC and Binutils can be built with -flto under MSYS, and they seem
> to work.

  Thanks for the report :)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #16 from davek at gcc dot gnu dot org  2010-02-03 14:46 -------
Created an attachment (id=19795)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19795&action=view)
Latest and updatest.

Updated:
- enable for mingw targets
- open files in binary mode in is_elf_or_coff
- bunch of commenting added.
- renamed the public lto_elf_* interface functions to lto_obj_*

Untested yet: about to take this for a bootstrap and test cycle.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #19636|0                           |1
        is obsolete|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #17 from davek at gcc dot gnu dot org  2010-02-03 14:48 -------
TO-DO: (additions invited, this is not by any means a complete list!)

- Add autoconfigury to ensure binutils supports .section directive alignment
syntax, and disable LTO if not.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
   Last reconfirmed|2010-01-18 16:18:01         |2010-02-03 14:48:26
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #18 from davek at gcc dot gnu dot org  2010-02-03 16:09 -------
Created an attachment (id=19797)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19797&action=view)
Bugfix of -respin-1

Opps!  Forgot to update the lto language hook definitions.  All fixed now.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #19795|0                           |1
        is obsolete|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #19 from davek at gcc dot gnu dot org  2010-02-04 17:12 -------
Created an attachment (id=19805)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=19805&action=view)
Further bugfix

fixed silly cut'n'pasto in the endianness layer which was truncating 4-byte
fields to 2 bytes.


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #19797|0                           |1
        is obsolete|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[steven at gcc dot gnu dot org]

------- Comment #20 from steven at gcc dot gnu dot org  2010-02-12 20:57 -------
What is the plan for this bug, fix it for GCC 4.5.0 or for later?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #21 from davek at gcc dot gnu dot org  2010-02-13 01:06 -------
(In reply to comment #20)
> What is the plan for this bug, fix it for GCC 4.5.0 or for later?

I don't really think I can argue that this is stage3 material, so the plan is
to fix it when trunk reopens for 4.6, and I have Richard G's
agreement-in-principle to then backport it from head to the 4.5 branch for
4.5.1


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[rguenth at gcc dot gnu dot org]

------- Comment #22 from rguenth at gcc dot gnu dot org  2010-02-13 10:23 -------
(In reply to comment #21)
> (In reply to comment #20)
> > What is the plan for this bug, fix it for GCC 4.5.0 or for later?
> 
> I don't really think I can argue that this is stage3 material, so the plan is
> to fix it when trunk reopens for 4.6, and I have Richard G's
> agreement-in-principle to then backport it from head to the 4.5 branch for
> 4.5.1

Yes, that was the idea.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #23 from sherpya at netfarm dot it  2010-04-09 03:55 -------
Created an attachment (id=20342)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20342&action=view)
updated for current trunk

obsoletes respin-3


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #24 from sherpya at netfarm dot it  2010-04-09 03:56 -------
Updated for current trunk, just compiled a cross gcc for mingw
I'll test if works


-- 

sherpya at netfarm dot it changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sherpya at netfarm dot it


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #25 from davek at gcc dot gnu dot org  2010-04-09 03:57 -------
(In reply to comment #24)
> Updated for current trunk, just compiled a cross gcc for mingw
> I'll test if works
> 

  Thank you!  Now that 4.6 is open I'll finish the work on this (the
autoconfery needs tightening up a bit) and submit it to the -patches list
sometime in the next 2-4 days.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #26 from sherpya at netfarm dot it  2010-04-09 04:02 -------
I'm getting a lot of:
lto1: warning: visibility attribute not supported in this configuration;
ignored
in the linking phase

using -fwhopr

and the link fails with a lot of:

../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.data+0xcb2c): undefined reference to
`_sigcheck_dummy.31957.3318'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.data+0x134cc): undefined reference
to `_sigcheck_dummy.31957.3318'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x15288): undefined reference
to `_deflate_stored.34680.3322'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x15294): undefined reference
to `_deflate_fast.34676.3326'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152a0): undefined reference
to `_deflate_fast.34676.3326'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152ac): undefined reference
to `_deflate_fast.34676.3326'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152b8): undefined reference
to `_deflate_slow.34672.3330'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152c4): undefined reference
to `_deflate_slow.34672.3330'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152d0): undefined reference
to `_deflate_slow.34672.3330'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152dc): undefined reference
to `_deflate_slow.34672.3330'
../../libclamav/7z.wpa.ltrans.o:7z.wpa.o:(.rdata+0x152e8): undefined reference
to `_deflate_slow.34672.3330'

using 

GNU ld (GNU Binutils) 2.20.51.20100408

the link is called by dllwrap:

i686-pc-mingw32-dllwrap -Wl,--enable-stdcall-fixup -fwhopr --def
../../contrib/msvc/libclamav.def --implib libclamav.dll.a -o libclamav.dll
___allobjs__ -lws2_32


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #27 from sherpya at netfarm dot it  2010-04-09 04:04 -------
these functions are static


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #28 from davek at gcc dot gnu dot org  2010-04-09 04:10 -------
(In reply to comment #27)
> these functions are static
> 

Hmm, some kind of inlining problem maybe?  There's a thread on the main GCC
list at the moment about problems with WHOPR, so I don't know to what extent
it's supposed to be working.

I also see that the command has ended up with -fwhopr but not -flto, I was
under the impression you had to use -flto always when using -fwhopr.

You appear to be testing by compiling libclamav, yes?  How did you configure
it?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #29 from sherpya at netfarm dot it  2010-04-09 18:30 -------
using -flto links, but looks like it links in a different way

while linking llvm part I get ice:

In member function 'RefineAbstractType':
lto1: internal compiler error: in input_gimple_stmt, at lto-streamer-in.c:1108
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://oss.netfarm.it/> for instructions.
lto-wrapper: /opt/mingw/bin/i686-pc-mingw32-g++ returned 1 exit status


clamav is not directly compilable with mingw
I maintain an unofficial port used in clamwin
I've made a simple set of makefiles (without deps by design)
to build the mingw version even in cross

git clone git://clamwin.git.sourceforge.net/gitroot/clamwin/clamav-win32.git

cd contrib/mingw
MINGW32_CROSS_PREFIX=i686-pc-mingw32- make
or something like
make llvm to make llvm dll (I've made a trick to have llvm jit optional)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #30 from sherpya at netfarm dot it  2010-04-09 19:48 -------
there is something odd.

with lto:
----------- SCAN SUMMARY -----------
Known viruses: 754681
Engine version: devel@clamwin MinGW - Apr  9 2010
Scanned directories: 1
Scanned files: 4402
Infected files: 0
Data scanned: 980.84 MB
Data read: 998.92 MB (ratio 0.98:1)
Time: 674.484 sec (11 m 14 s)

without:

----------- SCAN SUMMARY -----------
Known viruses: 754681
Engine version: devel@clamwin MinGW - Apr  9 2010
Scanned directories: 1
Scanned files: 4402
Infected files: 0
Data scanned: 980.84 MB
Data read: 998.92 MB (ratio 0.98:1)
Time: 419.938 sec (6 m 59 s)

a lot slower using lto?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #31 from davek at gcc dot gnu dot org  2010-04-10 16:20 -------
(In reply to comment #30)
> there is something odd.

> with lto:
> Time: 674.484 sec (11 m 14 s)

> without:
> Time: 419.938 sec (6 m 59 s)

> a lot slower using lto?

Is it possible you're just seeing the effects of file caching?  After the first
scan, you'd expect the second one to be faster.  Could you run them each say
three or five times in a row and see if it's just a side-effect of this?

Otherwise, it is of course certainly possible that there's some kind of bug in
LTO that leads to inadvertent pessimisation of code; it's a new feature after
all.  But it's surprising that it would show to such an extent in an
application like this, which I would have expected to be pretty much completely
I/O bound.

Thanks for doing all this testing, by the way; I haven't had time to try it out
on any big projects yet.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #32 from sherpya at netfarm dot it  2010-04-11 17:38 -------
I've repeated the test multiple times, I already done scan benchmarks before
I think I need to check if I have same results on linux


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[steven at gcc dot gnu dot org]

------- Comment #33 from steven at gcc dot gnu dot org  2010-04-11 22:59 -------
A common mistake is to not pass the optimizer flags properly to the linker.
There is a thread about that, too:
http://gcc.gnu.org/ml/gcc-patches/2010-04/msg00438.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #34 from sherpya at netfarm dot it  2010-04-11 23:58 -------
good point :) it should be written in caps, it's not common usage of linking
flags :)
anyway the reason is easy to understand

results aren't exactly as expected by at least not crazy :)

----------- SCAN SUMMARY -----------
Known viruses: 754681
Engine version: devel@clamwin MinGW - Apr 12 2010
Scanned directories: 1
Scanned files: 4402
Infected files: 0
Data scanned: 980.84 MB
Data read: 998.92 MB (ratio 0.98:1)
Time: 443.047 sec (7 m 23 s)


about the llvm dll I still get an ice

lto1: internal compiler error: in input_gimple_stmt, at lto-streamer-in.c:1108

          /* In case of type mismatches across units we can fail
             to unify some types and thus not find a proper
             field-decl here.  So only assert here if checking
             is enabled.  */
#ifdef ENABLE_CHECKING
          gcc_assert (tem != NULL_TREE);
#endif

any suggestion about submitting this bug? I suspect that intermediate file for
this report would become very large (normally the resulting binary is 8-9mb
large)

I've not tested natively but only a cross, but in my opinion your patch looks
ok, the produced code was always working


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[steven at gcc dot gnu dot org]

------- Comment #35 from steven at gcc dot gnu dot org  2010-04-12 10:40 -------
So if I understand correctly, the "state of things" at the moment is this:

Without LTO:
> Time: 419.938 sec (6 m 59 s)

with LTO incl linker flags:
> Time: 443.047 sec (7 m 23 s)

In other words, "with LTO" is ~6% slower than without.

You say that "results aren't exactly as expected" but I think there is really
not much you can expect.

First of all, as it is, in GCC 4.5 LTO/WHOPR does not do a whole lot at all. It
is really only inlining across translation unit boundaries.

Second, LTO doesn't miraculously increase the number of optimization
opportunities, and very often it makes things worse on register-starved
architectures like i686 (see
http://www.cs.rice.edu/~keith/512/Lectures/30IDFAO.pdf for folklore vs. fact in
re. interprocedural optimization).

Third, for C/C++ programs the source code is usually already organized in such
a way to avoid requiring a whole-program view of the program for the compiler
to optimize well. Think .h header files, static inline functions, by-value call
conventions, etc. And although I don't know clamav very well, I suspect it's
completely I/O bound so optimizing away memory latencies etc. (which is really
what LTO is mostly about) wouldn't help for clamav anyway.

That doesn't mean that optimizing LTO should slow down your program. It would
be interesting to see if it's somehow possible to identify the part of the
program that got slower.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #36 from davek at gcc dot gnu dot org  2010-04-12 13:30 -------
(In reply to comment #35)

> http://www.cs.rice.edu/~keith/512/Lectures/30IDFAO.pdf

  Thanks for the link, not just because it's full of intersting information,
but also because I now have a new candidate for
most-unlikely-combination-of-words-I-ever-thought-I'd-hear-used-in-a-real-sentence-and-agree-with-it:

"Fortran may be the best case scenario !"

(it's a pitched battle for the top spot between that and "Jar-Jar, you're a
genius!" right now)

:-)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[steven at gcc dot gnu dot org]

------- Comment #37 from steven at gcc dot gnu dot org  2010-04-12 15:58 -------
LTO for Mach-O is now being tracked in bug 43729.


-- 

steven at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  BugsThisDependsOn|                            |43729


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[ro at gcc dot gnu dot org]

------- Comment #38 from ro at gcc dot gnu dot org  2010-04-12 16:10 -------
Could be interesting for Tru64 UNIX, which uses ECOFF, too.


-- 

ro at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ro at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #39 from sherpya at netfarm dot it  2010-04-12 16:55 -------
(In reply to comment #35)
> So if I understand correctly, the "state of things" at the moment is this:
> 
> Without LTO:
> > Time: 419.938 sec (6 m 59 s)
> 
> with LTO incl linker flags:
> > Time: 443.047 sec (7 m 23 s)
> 
> In other words, "with LTO" is ~6% slower than without.
> 
> You say that "results aren't exactly as expected" but I think there is really
> not much you can expect.
> 
> First of all, as it is, in GCC 4.5 LTO/WHOPR does not do a whole lot at all. It
> is really only inlining across translation unit boundaries.
> 
> Second, LTO doesn't miraculously increase the number of optimization
> opportunities, and very often it makes things worse on register-starved
> architectures like i686 (see
> http://www.cs.rice.edu/~keith/512/Lectures/30IDFAO.pdf for folklore vs. fact in
> re. interprocedural optimization).
> 
> Third, for C/C++ programs the source code is usually already organized in such
> a way to avoid requiring a whole-program view of the program for the compiler
> to optimize well. Think .h header files, static inline functions, by-value call
> conventions, etc. And although I don't know clamav very well, I suspect it's
> completely I/O bound so optimizing away memory latencies etc. (which is really
> what LTO is mostly about) wouldn't help for clamav anyway.
> 
> That doesn't mean that optimizing LTO should slow down your program. It would
> be interesting to see if it's somehow possible to identify the part of the
> program that got slower.
> 

6% slower for a new work in progress feature is ok for me, while 60% not :)
clamav has a lot of io but also a lot of cpu so it can used for such tests
but definitively this patch is usable and could provide more feedbacks to lto


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #40 from davek at gcc dot gnu dot org  2010-04-13 05:58 -------
Submitted to -patches list at:
http://gcc.gnu.org/ml/gcc-patches/2010-04/msg00612.html


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #41 from davek at gcc dot gnu dot org  2010-04-13 06:01 -------
Thanks everyone for all the help with testing and validation :-)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[rguenth at gcc dot gnu dot org]

------- Comment #42 from rguenth at gcc dot gnu dot org  2010-04-23 16:10 -------
Fixed?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #43 from davek at gcc dot gnu dot org  2010-04-23 16:13 -------
(In reply to comment #42)
> Fixed?
> 

Still awaiting build system maintainer approval as per your request.  Ten days
is just on the lower margin of the range that I let a patch wait before pinging
it; I'll do so shortly.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[sherpya at netfarm dot it]

------- Comment #44 from sherpya at netfarm dot it  2010-04-24 02:46 -------
nice, it's a bit difficult to handle autoreconf in mingw :D, I need a specific
version on my linux box and then make a patch including configure and makefiles
changes :)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #45 from davek at gcc dot gnu dot org  2010-04-27 02:23 -------
Subject: Bug 42776

Author: davek
Date: Tue Apr 27 02:22:40 2010
New Revision: 158762

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=158762
Log:
ChangeLog:

        PR lto/42776
        * configure.ac (--enable-lto): Refactor handling so libelf tests
        are only performed inside then-clause of ACX_ELF_TARGET_IFELSE,
        and allow LTO to be explicitly enabled on non-ELF platforms that
        are known to support it inside else-clause.
        * configure: Regenerate.

gcc/ChangeLog:

        PR lto/42776
        * configure.ac (gcc_cv_as_section_has_align): Set if installed
        binutils supports extended .section directive needed by LTO, or
        warn if older binutils found.
        (LTO_BINARY_READER): New AC_SUBST'd variable.
        (LTO_USE_LIBELF): Likewise.
        * gcc/config.gcc (lto_binary_reader): New target-specific configure
        variable.
        * gcc/Makefile.in (LTO_BINARY_READER): Import AC_SUBST'd autoconf var.
        (LTO_USE_LIBELF): Likewise.
        * configure: Regenerate.

        * collect2.c (is_elf): Rename from this ...
        (is_elf_or_coff): ... to this, and recognize and allow i386 COFF
         object files in addition to ELF-formatted ones.
        (scan_prog_file): Caller updated.  Also allow for LTO info marker
        symbol to be prefixed or not by an extra underscore.

        * config/i386/t-cygming (winnt.o): Also depend on LTO_STREAMER_H.
        * config/i386/winnt.c: Also #include lto-streamer.h
        (i386_pe_asm_named_section): Specify 1-byte section alignment for
        LTO named sections.
        (i386_pe_asm_output_aligned_decl_common): Add comment.
        (i386_pe_maybe_record_exported_symbol): Allow decl arg to be NULL.

gcc/lto/ChangeLog:

        PR lto/42776
        * Make-lang.in (LTO_OBJS): Use LTO_BINARY_READER instead of
        hardcoding 'lto-elf.o'.
        ($(LTO_EXE)): Use LTO_USE_LIBELF instead of hardcoding '-lelf'.

        * lto-coff.h: New file.
        * lto-coff.c: Likewise.

gcc/testsuite/ChangeLog:

        PR lto/42776
        * lib/lto.exp (lto_prune_vis_warns): New function.
        (lto-link-and-maybe-run): Call it.


Added:
    trunk/gcc/lto/lto-coff.c
    trunk/gcc/lto/lto-coff.h
Modified:
    trunk/ChangeLog
    trunk/configure.ac
    trunk/gcc/Makefile.in
    trunk/gcc/collect2.c
    trunk/gcc/config.gcc
    trunk/gcc/config/i386/t-cygming
    trunk/gcc/config/i386/winnt.c
    trunk/gcc/configure
    trunk/gcc/configure.ac
    trunk/gcc/lto/ChangeLog
    trunk/gcc/lto/Make-lang.in
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/lib/lto.exp


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #46 from davek at gcc dot gnu dot org  2010-04-27 02:24 -------
Subject: Bug 42776

Author: davek
Date: Tue Apr 27 02:23:56 2010
New Revision: 158763

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=158763
Log:
Missing file from last commit!

ChangeLog:

2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * configure.ac (--enable-lto): Refactor handling so libelf tests
        are only performed inside then-clause of ACX_ELF_TARGET_IFELSE,
        and allow LTO to be explicitly enabled on non-ELF platforms that
        are known to support it inside else-clause.
        * configure: Regenerate.


Modified:
    trunk/configure


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #47 from davek at gcc dot gnu dot org  2010-04-27 02:25 -------
Subject: Bug 42776

Author: davek
Date: Tue Apr 27 02:24:51 2010
New Revision: 158764

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=158764
Log:
Missing changelog from last commit!

ChangeLog:

2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * configure.ac (--enable-lto): Refactor handling so libelf tests
        are only performed inside then-clause of ACX_ELF_TARGET_IFELSE,
        and allow LTO to be explicitly enabled on non-ELF platforms that
        are known to support it inside else-clause.
        * configure: Regenerate.

gcc/ChangeLog


2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * configure.ac (gcc_cv_as_section_has_align): Set if installed
        binutils supports extended .section directive needed by LTO, or
        warn if older binutils found.
        (LTO_BINARY_READER): New AC_SUBST'd variable.
        (LTO_USE_LIBELF): Likewise.
        * gcc/config.gcc (lto_binary_reader): New target-specific configure
        variable.
        * gcc/Makefile.in (LTO_BINARY_READER): Import AC_SUBST'd autoconf var.
        (LTO_USE_LIBELF): Likewise.
        * configure: Regenerate.

        * collect2.c (is_elf): Rename from this ...
        (is_elf_or_coff): ... to this, and recognize and allow i386 COFF
         object files in addition to ELF-formatted ones.
        (scan_prog_file): Caller updated.  Also allow for LTO info marker
        symbol to be prefixed or not by an extra underscore.

        * config/i386/t-cygming (winnt.o): Also depend on LTO_STREAMER_H.
        * config/i386/winnt.c: Also #include lto-streamer.h
        (i386_pe_asm_named_section): Specify 1-byte section alignment for
        LTO named sections.
        (i386_pe_asm_output_aligned_decl_common): Add comment.
        (i386_pe_maybe_record_exported_symbol): Allow decl arg to be NULL.


Modified:
    trunk/gcc/ChangeLog


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #48 from davek at gcc dot gnu dot org  2010-04-27 02:26 -------
Sorry, missed a couple of files the first time round and had to check them in
subsequently.  Oops.  *sheepish grin*


-- 

davek at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[ro at gcc dot gnu dot org]

------- Comment #49 from ro at gcc dot gnu dot org  2010-06-04 14:33 -------
Subject: Bug 42776

Author: ro
Date: Fri Jun  4 14:32:19 2010
New Revision: 160269

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=160269
Log:
        Backport from mainline:
        2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * lib/lto.exp (lto_prune_vis_warns): New function.
        (lto-link-and-maybe-run): Call it.

        2010-05-10  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

        * lib/lto.exp (lto_prune_vis_warns): Renamed to lto_prune_warns.
        Log text on entry and exit.
        Prune Sun ld warnings for common symbols with differing sizes.
        (lto-link-and-maybe-run): Change caller.

Modified:
    branches/gcc-4_5-branch/gcc/testsuite/ChangeLog
    branches/gcc-4_5-branch/gcc/testsuite/lib/lto.exp


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776

[Bug lto/42776] LTO doesn't work on non-ELF platforms.

[davek at gcc dot gnu dot org]

------- Comment #50 from davek at gcc dot gnu dot org  2010-06-14 10:38 -------
Subject: Bug 42776

Author: davek
Date: Mon Jun 14 10:38:18 2010
New Revision: 160722

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=160722
Log:
ChangeLog:

  Backport from mainline:
  2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * configure.ac (--enable-lto): Refactor handling so libelf tests
        are only performed inside then-clause of ACX_ELF_TARGET_IFELSE,
        and allow LTO to be explicitly enabled on non-ELF platforms that
        are known to support it inside else-clause.
        * configure: Regenerate.

gcc/ChangeLog:

  Backport from mainline:
  2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * configure.ac (gcc_cv_as_section_has_align): Set if installed
        binutils supports extended .section directive needed by LTO, or
        warn if older binutils found.
        (LTO_BINARY_READER): New AC_SUBST'd variable.
        (LTO_USE_LIBELF): Likewise.
        * config.gcc (lto_binary_reader): New target-specific configure
        variable.
        * Makefile.in (LTO_BINARY_READER): Import AC_SUBST'd autoconf var.
        (LTO_USE_LIBELF): Likewise.
        * configure: Regenerate.
        * collect2.c (is_elf): Rename from this ...
        (is_elf_or_coff): ... to this, and recognize and allow i386 COFF
         object files in addition to ELF-formatted ones.
        (scan_prog_file): Caller updated.  Also allow for LTO info marker
        symbol to be prefixed or not by an extra underscore.
        * config/i386/t-cygming (winnt.o): Also depend on LTO_STREAMER_H.
        * config/i386/winnt.c: Also #include lto-streamer.h
        (i386_pe_asm_named_section): Specify 1-byte section alignment for
        LTO named sections.
        (i386_pe_asm_output_aligned_decl_common): Add comment.
        (i386_pe_maybe_record_exported_symbol): Allow decl arg to be NULL.

gcc/lto/ChangeLog:

  Backport from mainline:
  2010-04-27  Dave Korn  <dave.korn.cygwin@gmail.com>

        PR lto/42776
        * Make-lang.in (LTO_OBJS): Use LTO_BINARY_READER instead of
        hardcoding 'lto-elf.o'.
        ($(LTO_EXE)): Use LTO_USE_LIBELF instead of hardcoding '-lelf'.
        * lto-coff.h: New file.
        * lto-coff.c: Likewise.

  2010-04-26  Dave Korn  <dave.korn.cygwin@gmail.com>

        * lto.h (lto_elf_file_open): Rename prototype from this ...
        (lto_obj_file_open): ... to this.
        (lto_elf_file_close): Likewise ...
        (lto_obj_file_close): ... and likewise.
        (lto_elf_build_section_table): Likewise ...
        (lto_obj_build_section_table): ... and likewise.
        (lto_elf_begin_section): Likewise ...
        (lto_obj_begin_section): ... and likewise.
        (lto_elf_append_data): Likewise ...
        (lto_obj_append_data): ... and likewise.
        (lto_elf_end_section): Likewise ...
        (lto_obj_end_section): ... and likewise.
        * lto.c (lto_file_read): Update references to the above.
        (lto_wpa_write_files): Likewise.
        (lto_read_all_file_options): Likewise.
        (read_cgraph_and_symbols): Likewise.
        * lto-lang.c (LANG_HOOKS_BEGIN_SECTION): Likewise.
        (LANG_HOOKS_APPEND_DATA): Likewise.
        (LANG_HOOKS_END_SECTION): Likewise.
        * lto-elf.c (lto_elf_file_open): Rename from this ...
        (lto_obj_file_open): ... to this, updating any references.
        (lto_elf_file_close): Likewise ...
        (lto_obj_file_close): ... and likewise.
        (lto_elf_build_section_table): Likewise ...
        (lto_obj_build_section_table): ... and likewise.
        (lto_elf_begin_section): Likewise ...
        (lto_obj_begin_section): ... and likewise.
        (lto_elf_append_data): Likewise ...
        (lto_obj_append_data): ... and likewise.
        (lto_elf_end_section): Likewise ...
        (lto_obj_end_section): ... and likewise.


Added:
    branches/gcc-4_5-branch/gcc/lto/lto-coff.c
    branches/gcc-4_5-branch/gcc/lto/lto-coff.h
Modified:
    branches/gcc-4_5-branch/ChangeLog
    branches/gcc-4_5-branch/configure
    branches/gcc-4_5-branch/configure.ac
    branches/gcc-4_5-branch/gcc/ChangeLog
    branches/gcc-4_5-branch/gcc/Makefile.in
    branches/gcc-4_5-branch/gcc/collect2.c
    branches/gcc-4_5-branch/gcc/config.gcc
    branches/gcc-4_5-branch/gcc/config/i386/t-cygming
    branches/gcc-4_5-branch/gcc/config/i386/winnt.c
    branches/gcc-4_5-branch/gcc/configure
    branches/gcc-4_5-branch/gcc/configure.ac
    branches/gcc-4_5-branch/gcc/lto/ChangeLog
    branches/gcc-4_5-branch/gcc/lto/Make-lang.in
    branches/gcc-4_5-branch/gcc/lto/lto-elf.c
    branches/gcc-4_5-branch/gcc/lto/lto-lang.c
    branches/gcc-4_5-branch/gcc/lto/lto.c
    branches/gcc-4_5-branch/gcc/lto/lto.h


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42776